[Geomag-data] instability of USGS Dst Index in recent weeks

[W. Kent Tobiska]

Thanks for the explanation of this, Carol. We will consider this issue for the USGS Dst.

Best regards,
Kent

On Feb 23, 2017, at 10:35 PM, Finn, Carol <cafinn at usgs.gov> wrote:

> The USGS San Juan (SJG) magnetic observatory, one of four international observatories used to generate Dst, recently experienced a catastrophic failure. The data generated during this failure was not blocked from transmission, and still resides in USGS digital archives, where it contributes to the USGS real time Dst index calculations. The SJG observatory has been repaired, and is producing reliable data once again, but the algorithm used to remove its slow secular and periodic (daily) variations will remain impacted as long as the faulty data resides within the algorithm's processing window. This will improve as more new data are collected, but it will not disappear entirely until this 1 year window has passed.
> 
> Some users have asked why we cannot simply "null out" the bad data, which would likely reduce, if not completely eliminate the problems with SJG and Dst. The fact of the matter is that USGS Geomagnetism Program data stewardship policies require all "raw" data to be retained, un-modified from its original state. This includes products derived directly from this "raw" data, including our real time Dst. Of course these policies do not prevent algorithmic data "cleaning", or other methods designed to make the algorithm more robust to real time data issues. But the current USGS real time Dst algorithm does not attempt to filter out bad data, and because it is legacy software no longer supported by its original author, it cannot be easily modified to do so.
> 
> The latter has been a concern for some time, and a replacement algorithm more suited to real time operations has been developed, and was actually in the process of validation at the time of the SJG failure. We have re-prioritized its deployment schedule, and hope to soon have a new-and-improved real time USGS Dst index to share with the geomagnetism community.
> _______________________________________________
> Geomag-data mailing list
> Geomag-data at geohazards.usgs.gov
> https://geohazards.usgs.gov/mailman/listinfo/geomag-data

**********************************************
W. Kent Tobiska
  President, Space Environment Technologies
  Chief Scientist  (Space Weather Division)
  1676 Palisades Dr., Pacific Palisades, CA 90272
  1-310-573-4185 (office) 1-310-454-9665 (fax)
  ktobiska at spacenvironment.net
  http://SpaceWx.com
**********************************************
Enabling human evolution into space
     with advanced space weather
         products and services
**********************************************




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://geohazards.usgs.gov/pipermail/geomag-data/attachments/20170224/7cc4b958/attachment-0001.html>