<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"><meta name="Generator" content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang="EN-US" link="blue" vlink="purple"><div class="WordSection1"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Philip,</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Firstly the issue with RTP/RTPD links sleeping due to outage. This points to something at the RTPD server end. If a RTP/RTPD is declared down the 130 will go into a sequence of sending server discovery packets to the RTPD host address every 6-8secs, for a 300sec on/120sec RTP sleeping, cycle until the link is re-established. RTPD when sensing this unconditional sync, also known as server discover, will start to negotiate with the units RTP to bring the link up. I would need to look at the RTPD log file to see if the unconditional syncs are coming into RTPD and if so what RTPD responds with. I suspect there is a time out in the local firewall or router handling the traffic from the 130 units to the RTPD server. And sending the Das discovery resets this time out. A server discovery from RTPD to a 130 is different from what RTPDID sends out and what RTPD sends to a DAS if it receives an unconditional sync from it. If you could send a RTPD log file I could at least confirm that server discoveries from the 130’s are being seen by RTPD.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Secondly your issue with 99minutes of buffered data. The firmware in the 130 is written in such a way that when the RTP/RTPD is declared down the RTPD thread data will be saved to RAM for the thread’s TOSS threshold. In your case 99mins. However if the link remains down for more than the TOSS threshold this RTPD thread data since the link down declaration is all deleted from. Again in your case this happens at 99mins. REF TEK’s logic to do this has been explained to allow RAM to be freed up to handle other thread data link Disk Thread because it is unlikely the link will re-establish if the TOSS threshold has been met and this old RTPD thread data will continue to get older etc, and therefore of lesser value when the link is re-established. If you find you have link outages lasting on average 110mins then simply increase the TOSS threshold so most data will be recovered.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Again please send me a RTPD log file that has a time window of known 130 to central link stability but no RTP/RTPD connection as well as the part of the log that shows before and after result of the user issued server discovery.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks,</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Ian Billings</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Field Technician</span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">REF TEK – A Division of Trimble Navigation</span></b></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><a href="http://support.reftek.com/">http://support.reftek.com</a></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Skype ian_billings1</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">PH 214 440 1265</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">PH 214 440 1289 (Direct)</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> ANSS-netops [mailto:<a href="mailto:anss-netops-bounces@geohazards.usgs.gov">anss-netops-bounces@geohazards.usgs.gov</a>] <b>On Behalf Of </b>Philip Crotwell<br>
<b>Sent:</b> Friday, February 22, 2013 10:41 AM<br><b>To:</b> <a href="mailto:anss-netops@geohazards.usgs.gov">anss-netops@geohazards.usgs.gov</a><br><b>Cc:</b> Thomas J. Owens<br><b>Subject:</b> [ANSS-netops] reftek and data stoppages</span></p>
<p class="MsoNormal"> </p><div><div><div><div><div><div><div><div><div><p class="MsoNormal"> </p></div><p class="MsoNormal" style="margin-bottom:12.0pt">Hi all</p></div><p class="MsoNormal" style="margin-bottom:12.0pt">We have four stations with reftek 130s on cell modems going into earthworm via rtpd. I recently moved my server to new hardware and rediscovered an old problem. About every day or two some of the stations stop sending data even though the link is ok. I don't know what the initial cause is, maybe the cell link briefly died, but at the time we check on things, the cell link is fine but data is just not flowing.</p>
</div><p class="MsoNormal" style="margin-bottom:12.0pt">The odd thing is the clicking the "das discovery" button in the web admin tool RTCC causes all the stations to start flowing again. The rediscovery part is that some years ago when we first noticed this, I put in a cron job to hit the "das discovery" url once every 15 minutes and the problem went away. Given limited brain cells, I promptly forgot about it. Not until I switched server machine, and forgot to transfer the cron job, did I remember the issue.</p>
</div><p class="MsoNormal" style="margin-bottom:12.0pt">It is very puzzling to me that if getting the stations back on line is as easy as clicking a url, then why in the world can't rtpd do it itself!?!?? Have any of you seen this issue? Any suggestions on ways to deal with it other than a cron based das discovery? I should say we run a mixed network with other statins using either q330s or guralps, and only the refteks seem to have trouble noticing that the cell link is working.</p>
</div><div><p class="MsoNormal" style="margin-bottom:12.0pt">One other puzzle is that my understanding is that the rt130s will cache up to 99 minutes of data in the case of a lost connection. My experience is that you get the benefit of the cache only in cases of the outage lasting less than 99 minutes. If the outage is longer, then when the link comes up the rt130 starts sending real time data and never sends the previous 99 minutes. If however the outage is less than the cache time, it will start sending the old cached data first. Seems weird that a 98 minute outage results in no data loss, but a 100 minute outage results in a 100 minute data loss.</p>
</div><p class="MsoNormal" style="margin-bottom:12.0pt">We have recent, but not the absolute latest versions of firmware, so I should probably upgrade those just in case. We have stations showing this issue with firmware at recent as 3.3.1 and I don't see anything in the release notes that would suggest newer firmware addresses this. RTPD on the server is the latest version, 2.1.9.0b.</p>
</div><p class="MsoNormal" style="margin-bottom:12.0pt">Here is some output of me running rtpid around the time I hit the "das discovery" button. I hit the button at 11:29:00 and all the stations had come back to life and sending data within 12 seconds of the discovery action.</p>
</div><p class="MsoNormal">thanks,</p></div><p class="MsoNormal">Philip</p><div><p class="MsoNormal" style="margin-bottom:12.0pt"><br>2013:053-11:27:56 earthworm rtpid[3545] RTPID version 2.1.0.0<br>2013:053-11:27:56 earthworm rtpid[3545] Options:<br>
2013:053-11:27:56 earthworm rtpid[3545] Host = localhost<br>2013:053-11:27:56 earthworm rtpid[3545] Port = 2543<br>2013:053-11:27:56 earthworm rtpid[3545] Retry = nonfatal<br>2013:053-11:27:56 earthworm rtpid[3545] Log file = rtpid.log<br>
2013:053-11:27:56 earthworm rtpid[3545] Verbose = FALSE<br>2013:053-11:27:56 earthworm rtpid[3545] Timeout = 60<br>2013:053-11:27:56 earthworm rtpid[3545] Attempts = 9999<br>2013:053-11:27:56 earthworm rtpid[3545] Attempting connection: localhost:2543<br>
2013:053-11:27:56 earthworm rtpid[3545] Successful connection: localhost:2543</p></div><div><p class="MsoNormal"> ---- hit "DAS-DISCOVERY" at 11:29:00 ----</p></div><div><p class="MsoNormal" style="margin-bottom:12.0pt">
<br>2013:053-11:29:02 earthworm rtpid[3545] Unit A898 detected<br>2013:053-11:29:03 earthworm rtpid[3545] Unit A064 detected<br>2013:053-11:29:06 earthworm rtpid[3545] Unit A872 detected<br>2013:053-11:29:12 earthworm rtpid[3545] Unit A900 detected</p>
</div></div></div></body></html>