Due to large amount of depth and breadth of data Insight works with, the default behavior for all data sources are nightly and weekly refreshes occurring between 12 am and 1 am Pacific.
- Daily refreshes are incremental meaning Insight compares what it has to date and uses the last most recent data record as its starting point to check for new data. For example, if Insight had up to row ID 250,000 for a database table and refreshed that night, the most current row ID for that table might be 300,000. When Insight refreshes that data it will start from row ID 250,001 and fetch up to 300,000. This is more effective than pulling in the entire valid set each night.
- Weekly refreshes occur every Saturday and perform a full data refresh. This is done to prevent data loss from occurring in the event data is retroactively added and the increment logic does not detect it. The only way to do this would be to directly inject rows with older record IDs than what is current. For example, if the highest current row ID is 250,000 and rows 25,000 and 25,001 were inserted manually, the refresh logic would not see these "new" records on the nightly incremental refresh.
Insight does have live connection capability, but does not use it by default. When using live connections, Insight must directly query the relevant data sources, which may be very large depending on the criteria. If you believe you have a use case for reporting or data that is mission critical and requires live updating, please send your inquiry to support@noctel.com.
What Happens If a Data Refresh Fails?
In the event of a data refresh failure, NocTel Insight's engineering team is notified automatically of the occurrence. Once notified, the team begins debugging and implementing fixes to correct. When these changes are made, the affected data source(s) are manually refreshed to qualify the effectiveness of the changes. In this situation, you would not need to wait for the next nightly or weekly full refresh to occur before fresh data is available.
Should I Report Any Cases Where the Most Current Data is Missing?
In most cases the NocTel Insight team does not recommend raising a support ticket for missing data. This is due to the fact the team is already aware of any failed data source refreshes. If you do feel it is appropriate to notify the team of such issues relating to the freshness of available data, please take into account the timing of the inquiry - it may be possible the nightly or weekly refresh has not yet completed.
Rather than inquiring about missing data, a better question to consider is "Is the default refresh interval sufficient for my organization?" If the default refresh intervals are too infrequent, please notify the team of this and state your use case and requirements. The Insight team may be able to create a more frequent refresh interval for your organization.
My Organization Requires a More Frequent Refresh Interval
Contact us at support@noctel.com with your need. In most cases with new customer organizations reporting requirements will be laid out that typically define the necessary data refresh intervals if the default rates are insufficient.