[Geomag-data] instability of USGS Dst Index in recent weeks

[Finn, Carol]

The USGS San Juan (SJG) magnetic observatory, one of four international
observatories used to generate Dst, recently experienced a catastrophic
failure. The data generated during this failure was not blocked from
transmission, and still resides in USGS digital archives, where it
contributes to the USGS real time Dst index calculations. The SJG
observatory has been repaired, and is producing reliable data once again,
but the algorithm used to remove its slow secular and periodic (daily)
variations will remain impacted as long as the faulty data resides within
the algorithm's processing window. This will improve as more new data are
collected, but it will not disappear entirely until this 1 year window has
passed.

Some users have asked why we cannot simply "null out" the bad data, which
would likely reduce, if not completely eliminate the problems with SJG and
Dst. The fact of the matter is that USGS Geomagnetism Program data
stewardship policies require all "raw" data to be retained, un-modified
from its original state. This includes products derived directly from this
"raw" data, including our real time Dst. Of course these policies do not
prevent algorithmic data "cleaning", or other methods designed to make the
algorithm more robust to real time data issues. But the current USGS real
time Dst algorithm does not attempt to filter out bad data, and because it
is legacy software no longer supported by its original author, it cannot be
easily modified to do so.

The latter has been a concern for some time, and a replacement algorithm
more suited to real time operations has been developed, and was actually in
the process of validation at the time of the SJG failure. We have
re-prioritized its deployment schedule, and hope to soon have a
new-and-improved real time USGS Dst index to share with the geomagnetism
community.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://geohazards.usgs.gov/pipermail/geomag-data/attachments/20170223/699cb72e/attachment.html>