Apr 23

Microsoft Azure SQL Server Usage and DTU Limit Resource Graphs are Confusing

microsoft-azure-h60 SQLServerText

Microsoft Azure lets you easily create and deploy enterprise quality SQL Server on the cloud and scale it to suit your application’s needs. From the SQL Server database’s Azure dashboard, you can see the Database Transaction Unit (DTU) usage against the specified DTU limit for the database.

One Hour Usage Graph

This is what we saw for usage over one hour. The cyan line across the top is the DTU limit. The dark blue line is the DTU used. The limit is what you pay, so it’s important to scale it to what the application needs.

monitor-hour

One Hour Azure SQL Server DTU use versus limit

While everything seemed fine at the weekly level, looking at the hourly graph gave us a shock. It looks like the database is maxed out for most of the hour. It seems conclusive that we need to increase our DTU level.

65 Minute Graph

But when we set the range to 65 minutes and saw this:

monitor-65-minute

65-minute Azure SQL Server DTU use versus limit

These are completely different displays of the same period of time. The 65 minute graph never hits the maximum DTU. What’s going on?

Visit our page Monitoring SQL Server Usage on Microsoft Azure and Setting DTU Limits for an explanation.

Apr 12

What Happened When We Created a Facebook App for Social Network Analysis

facebook-medFacebook and Mark Zuckerberg are getting blamed for a large number of issues from promoting fake news, election fraud, mishandling user data, and profiting from selling user data.

While some of that may be true, the Facebook security breach is actually a violation of Facebook API licensing rules by the people who used it. Facebook provided the data and encouraged developers like us to create innovative solutions for the Facebook ecosystem. They weren’t selling the data.They weren’t even charging us to use it.

Our Facebook App with Social Network Analysis and Maps

In 2010, we created a Facebook application using our Sentinel Visualizer technology to perform Social Network Analysis (SNA) based on a user’s friends’ friends. It would automatically cluster friends so you could quickly see their groups (high school, college, work, family, in-laws, clubs, etc.).

Facebook Social Network Analysis App of Clustered Friends

Each box (picture) was one of your friends, and you could move them around the network, hover over them to get their info, or click on them to go to their page.

We also plotted friends on a Microsoft Bing Map making it easy to see who were near you or where you were visiting.

Plotting Your Friends' Locations on a Map

 

We launched our free Sentinel Visualizer Facebook App to a limited number of users and it started to gain followers. People were amazed to see which of their friends knew each other. The application started to go viral. We were having trouble supporting the traffic.

Not Allowed to Save Facebook Data

One of the things developers couldn’t do was to save Facebook’s data. All we collected were the user names and email addresses people provided when they registered our program. Unfortunately, other developers didn’t abide by Facebook’s terms and the data improperly got to Cambridge Analytica and others.

Facebook Stopped Making the Data Available

Our app ceased to work when Facebook limited their APIs and prevented our ability to get to the list of your friends’ friends among your network.

It’s not entirely Facebook’s fault for trying to spur innovation by sharing their data for free. Some developers violated the trust Facebook gave them.

The Full Story

Here’s our new web page describing our experience in detail:

Mar 14

Create Surveys with Excel Online hosted in Office365

excel-icon-lgYou can quickly create surveys with Excel and make them available online with all results saved in the spreadsheet hosted in Office365. Just create an Excel spreadsheet in Office365 and choose the Survey item on the ribbon to design it.

A link is provided to share with people to respond. They don’t need to have Excel or Office365. Just a web browser. Try our sample by clicking the image below:

survey

Shorten the URL

Excel provides a very long URL to share your survey. By using a site like bit.ly, you can create a shorter one to email to your contacts. This is important to avoid problems when URLs get word wrapped in emails. For instance, this survey is shortened to: https://bit.ly/2qmSweG

Results in Your Spreadsheet

The results go directly into your spreadsheet with each submission as a separate row:

response

Having the ability to create a survey and see the results in a spreadsheet in minutes, is a terrific tool to help your projects!

 

Jan 05

Sean Hannity Radio Show Interview on Julian Assange, WikiLeaks, Russian Hacking, and Cyber Warfare

sean-hannity-radioBackground

The day after an amazing personal interview of Julian Assange by Sean Hannity aired on his TV show, FMS President Luke Chung was invited to discuss the related technology on his radio show.

Hannity traveled to London to interview Julian Assange at the Ecuadorian Embassy where he’s seeking asylum. They discussed an overview of Assange’s role as founder of WikiLeaks, and their obtaining and publishing the emails from the Democratic National Committee the weeks before the US Presidential election. Some people attribute Hillary Clinton’s loss to the revelations in those emails especially from John Podesta, the former White House Chief of Staff and Chairman of the Clinton campaign. They are also accusing the Russians for hacking (stealing) and providing the data to Assange so Donald Trump could win the election.

Radio Show

On January 4, 2017, I was on the radio show with Sean Hannity and Brigadier General Eli Ben Meir, former Israeli Military Intelligence chief. The three of us discussed the WikiLeaks disclosures. I commented specifically on:

  • Cyber attacks and the security breach at OPM disclosed non-classified government employees and by omission who were covert at American Embassies globally.
  • Noting Julian Assange’s careful word choices to exclude Russia as source without excluding them as the ultimate source of his sources.
  • The need for WikiLeaks’ to keep their sources confidential and how they amplified the data from Bradley Manning and Edward Snowden.
  • Different approaches to preventing cyber attacks depending on the cause.
    “It’s one thing when someone steals your car because they broke into it. It’s another thing when someone steals your car because you left your keys in the ignition.”

Here’s the audio of the show:

My segment starts at the 6:50 mark. Sean and General Meir speak first, then I start around 9:25. Final comments at 14:15 and it wraps up by 14:50.

Additional Issues

Only a limited amount of information can be discussed in such a short interview. Some additional issues to consider are:

Data Security

Securing data over the internet and inside organizations is very challenging. Threats may come from:

  • External hacks that need to be monitored and defeated
  • Internal people who unintentionally leave the front door unlocked
  • Internal people who intentionally leak information

Different solutions are required for each type of threat. Some are at the software vendor, design, and developer level, while others involve end-user training, background checks, and monitoring.

Applications can be built so that simply disclosing a user name and password doesn’t compromise the whole system by require two-factor authentication and registering devices that can use those credentials.

Unfortunately, many systems were built well before today’s cyber threats existed. The cost of making those systems more secure without breaking their existing functionality will be daunting and expensive. In many cases, the original source code, development environment and/or vendor are long gone, so the only option is to replace them which is also very expensive and time consuming.

Julian Assange and WikiLeaks are Not Heroes

We need to keep in mind that the WikiLeaks exposed top secret US information by publishing the disclosures from Manning and Snowden. Lives were put at risk and lives may have been lost because of those publications. The Arab Spring was inflamed in part by the disclosure of diplomatic communications and one could argue the human tragedy in Syria is tied to this as well. While Republicans are celebrating and defending Assange and WikiLeaks now for the DNC emails, the tables may turn very quickly.

Data That’s Not Exposed May be More Dangerous

While many are focused on the DNC emails, it’s not unreasonable to assume the people who hacked that also got the RNC emails. Data can be power, and in the wrong hands, data can be used for nefarious purposes such as blackmail.

If the RNC data were compromised, we should be extremely worried if the hackers discovered it was more valuable to keep private than public. Whether they use it directly or sell it to another party or country, the information can make victims puppets by threatening the exposure of their personal data. It’s not uncommon during E-discovery of an email server to discover all sorts of inappropriate language, behavior and activity conducted by individuals in an organization. Disclosures of affairs, homosexual activity, underage sex, bribery, unethical business dealings, breeches of confidentiality, collusion, and actual crimes are often found in email threads and can be used for blackmail.

Additional Resources

Jul 01

Microsoft Office365 Exchange Online Service Performance Degradation and SMTP Problems

office365A few years ago, we migrated our email service to Microsoft’s Office365 cloud service. Overall, it’s been very reliable and eliminated the challenges we had hosting Exchange ourselves. It let us get to our emails using Outlook installed on Windows, any internet browser, and smartphones. Office365 also offered other Office product online (Access Web Apps, Excel, Word, etc.), SharePoint and OneDrive Business.

Unfortunately, on the morning of June 30th, we discovered:

  • Delays sending and receiving emails
  • Some emails were bouncing back from recipients who couldn’t validate our Office365 Exchange Server’s SMTP (protection.outlook.com) with our domain name. That meant the Exchange SMTP server was no longer considered a trusted sender of emails from the @fmsinc.com domain.
  • Our use of the Office365 SMTP server to send emails with our Total Access Emailer product was also failing to authenticate against the server

The problems began the evening before. Needless to say, we aren’t happy about this experience which impacted us and our clients using Office365. Reports are that it affects Office365 customers across North America.

exchangeContacting Microsoft, they confirmed problems with the health of their Office365 Exchange Server. Throughout the day, problems lessened but persisted. We hope the problems are resolved soon and that we’ll understand what went wrong once we overcome the immediate crises.

These are the reports we’ve received from Microsoft. We’ll keep you updated as we learn more:


Exchange Online Service Degraded

This is what the Office365 Admin portal shows for Service Health:

Office365Issues

Office365Health


EX71628 – E-Mail and calendar access – Restoring Service

Jun 29, 2016 12:11 PM

CURRENT STATUS

Our investigation determined that an existing transport feature which is designed to expedite the delivery of email messages became degraded, which caused impact to email delivery for a subset of users. We’re bypassing the affected feature to restore service

User Impact

Users may be unable to send email messages through the Exchange Online service. Email messages may appear to be stuck in the Drafts or Outbox folders.

Scope of Impact

A few customers have reported this issue, and our analysis indicates that for most customers, it’s unlikely that many users would report impact related to this event.

  • Start Time: Thursday, June 23, 2016, at 3:00 PM UTC

Preliminary Root Cause

An existing transport feature that is designed to expedite the delivery of email messages became degraded, which caused impact to email delivery for a subset of users

EX71628 – E-Mail and calendar access – Extended recovery

Jun 30, 2016 2:18 PM

Current Status

We’ve developed an additional fix to address the underlying cause of the issue. We’re preparing to deploy the fix to the affected environment to ensure that the issue does not reoccur.

User Impact

Users may be unable to send email messages through the Exchange Online service. Email messages may appear to be stuck in the Drafts or Outbox folders.

Scope of Impact

A few customers have reported this issue, and our analysis indicates that for most customers, it’s unlikely that many users would report impact related to this event.

  • Start Time: Thursday, June 23, 2016, at 3:00 PM UTC

Preliminary Root Cause

An existing transport feature that is designed to expedite the delivery of email messages became degraded, which caused impact to email delivery for a subset of users.

Next Update by: Saturday, July 2, 2016, at 7:00 PM UTC


EX71674 – E-Mail timely delivery – Service restored

Jun 30, 2016 7:35 PM

Final Status

We’ve confirmed that the remaining message queues have now drained after implementing a configuration change to optimize message filtering.

User Impact

Users were experiencing delays when sending and receiving email messages. Affected users may have received Non-Delivery Reports (NDR) when sending email messages.

Scope of Impact

Customer reports indicated that many users likely experienced impact related to this event. Our analysis indicates that this issue may potentially have affected any of your users attempting to send or receive mail.

  • Start Time: Thursday, June 30, 2016, at 2:30 PM UTC
  • End Time: Thursday, June 30, 2016, at 11:30 PM UTC

Preliminary Root Cause

The infrastructure responsible for processing Exchange Online Protection (EOP) message filtering became degraded.

Next Steps

  • We’re analyzing performance data and trends on the affected systems to help prevent this problem from happening again.
  • We’re reviewing our code for optimizations and automated recovery options.
  • We’ll publish a post-incident report within five business days.

EX71674 – E-Mail timely delivery – Service restored

Jul 1, 2016 12:08 AM

Final Status

We’ve rolled out the fix and confirmed that service is restored. Any meeting requests created during the outage will need to have the conference room calendar removed and readded to book the room.

User Impact

Users that attempted to create a meeting request with a conference room calendar were unable to successfully book a conference room. This lead to conference rooms being booked by multiple resources.

Scope of Impact

A few customers reported this issue, and our analysis indicated that this may have affected any users attempting to use this feature.
  • Start Time: Monday, June 27, 2016, at 6:00 PM UTC
  • End Time: Friday, July 1, 2016, at 2:54 AM UTC

Preliminary Root Cause

A recent update affected the ability for calendar invite requests to successfully book conference rooms.

Next Steps

  • We’re reviewing our deployment and provisioning procedures to help prevent this kind of problem in the future.
  • We’ll publish a post-incident report within five business days.