About Luke Chung-FMS

Luke Chung is the President and Founder of FMS, Inc. FMS has provided professional software solutions for 30+ years through its commercial products, Sentinel Visualizer link analysis product, and database consulting services for Windows, web and mobile platforms. FMS has tens of thousands of customers in over 100 countries. More information
Dec 30

Government Shutdown’s Impact on Contractors and Employees

Contractors Face Considerable Financial Risk

With Congress and President Trump at a budget impasse, funding for significant portions of the US Federal government ended midnight Friday, December 18, 2018. Historically, government employees are not paid on time but are retroactively paid after the new budget is implemented. The same is not true for government contractors.

Government contractors compete against each other to provide a wide range of services such as computer programming, construction, security guards, cafeterias, janitorial services, technical experts, maintenance workers, and much more. When the government shuts down, contractor staff are not needed or can’t do their work. But they are real companies and people who suffer. Long-term, it’s bad for US taxpayers.

Our Contract with Amtrak is Suspended

We have a government contract with Amtrak which was suspended by this letter:

This contract provides help for Microsoft Access database programming and is relatively small. We do not have dedicated staff for the contract, so we are not really affected as our people will work on other projects.

Unfortunately, we have colleagues who are government contractors or in businesses more dependent on the government who are significantly impacted by the shutdown. This is especially true for companies who provide staff to work at government facilities that are now closed. Most are hesitant to speak publicly about their experiences in fear their government clients would be offended, so we’d like to share our experiences and theirs.

Very Tough Situation and Decisions for Government Contractors and Their Employees

Since government contractors are unlikely to be repaid when the budget is finally resolved, government contractors with dedicated staff for those contracts need to decide whether to pay their employees for time they never expected to not charge to the government. The impact is somewhat offset by the Christmas and New Year’s holidays, and maybe they can force employees to take vacation days. But as the shutdown lasts longer, contractors will have to pay their employees from their own funds. Not paying them runs the risk that the employees leave and won’t be available once the budget is resolved. One cannot expect employees to remain loyal if they aren’t paid. Meanwhile, employees are dusting off their resumes and seeking alternative employers.

Small businesses are especially vulnerable since many contracts have small profit margins, so without significant cash and lines of credit, a cash flow problem can quickly bankrupt a company. Even generous owners who try to do the “right thing”, may not be able to if their cash is depleted. That would lead to fewer government contractors in the future and higher costs to taxpayers.

A friend of mind who owns a firm completely focused on servicing the federal government shared:

“As contractors, our employees still get paid and yet we are unable to bill. Cash is king and small companies like mine live and die by cash. This is the worst.”

Media Coverage

On December 26th, we sent a Twitter message with the letter we received from Amtrak and the sympathy we feel for government contractors and their employees.

That led to our inclusion in a Washington DC News article on WUSA Channel 9 (CBS) by John Henry that evening. A video of the coverage with our Amtrak letter and insight are in this article: #ShutdownStories: Government contractors tackle life without pay during partial shutdown

On December 27th, I was interviewed in the Huffington Post article by Sarah Ruiz-Grossman about the challenges facing contractors entitled
Shutdown Leaves Government Contractors Without Work And Likely No Back Pay.

“Business owners who have [dedicated] staff are making the decision: Do we pay people or not, even though we won’t get paid by the government? Either you force people to take vacation, or you pay them, to be a good company ― but depending how long it drags on, one may not have a choice.”

On December 27th, Bridget Johnson included us in her Homeland Security Today article:
Federal Employees, Contractors Tweet Worries with #ShutdownStories

Self-Inflicted Wound that Should be Resolved Promptly

Eventually the budget will be resolved. We hope the parties come to their senses and do it sooner rather than later because innocent people who’ve dedicated their firms and lives to providing a professional service to US taxpayers are at risk and feeling real pain. That’s terrible for them now and our country long-term.

May 10

Remote Desktop Authentication Error Has Occurred. The function requested is not supported. CredSSP Workaround

Remote Desktop Connections Fail

Starting May 9, we received many reports of Remote Desktop connections failing globally. Users received error messages like this when they tried to remote to machines they connected to successfully for a long time:

Remote Desktop Connection Error

An authentication error has occurred.
The function requested is not supported

Remote computer: <computer name>
This could be due to CredSSP encryption oracle remediation.
For more information, see https:/go.microsoft.com/fwlink/?linkid=866660

The link goes to this page, https://support.microsoft.com/en-us/help/4093492/credssp-updates-for-cve-2018-0886-march-13-2018, and explains the Credential Security Support Provider protocol (CredSSP). It offers extensive information on a series of updates since March 2018. It recommends some steps but isn’t very clear what those changes are nor whether those changes are needed to be made by network administrators globally via group policies, or group policies on every PC and VM.

Caused by a Microsoft Security Patch

The Microsoft Security patch issued on Tuesday, May 8th triggered the problem by setting and requiring remote connections at the highest level (CredSSP Updates for CVE-2018-0886)::

Security update deployment information: May 08, 2018

It changed the default setting from Vulnerable to Mitigated which means that any PC using CredSSP is not be able to use insecure versions. If your PC received the May update but the target PC hasn’t implemented the CredSSP update, the PC receives the error message when it tries to connect to that PC.

The automatic Windows patch to raise the security level is not implemented if the PC doesn’t allow automatic updates. This mismatch between the implementation of a security requirement (which is not optional) without the corresponding automatic update may be the source of this problem.

However, there are many situations such as development, testing, build, staging, and deployment environments which require a stable environment that would be destroyed by automatic Windows updates.

We continue to research this.

Symptoms

The symptoms are rather strange because we found that some machines successfully connected while others didn’t.

For instance, we had a Windows 7 machine that hosted Remote Desktop. A Windows 7 PC had no problem connecting to it, but the same user connecting from a Windows 10 machine failed when that was never an issue before and the host machine allowed remote connection for years.

There are also reports of problems with Windows 10 machines connecting to Windows 10 machines, and people locked out of their Azure VMs.

Workaround Solution

One could rollback the security update, but rather than risking other security problems, there’s a quick fix.

Simply adjust the Remote Desktop settings on the host machine to a lower security level. From File Explorer, choose Computer, right-click and select Properties, then click Change Settings, and go to the Remote tab.

From Windows 10, uncheck the option to “Allow connections only from computers running Remote Desktop with Network Level Authentication (recommended)”:

From Windows 7, it’s setting the option to the Less Secure option rather than More Secure:

Once these are set, users can remote to the machine again.

Microsoft Comment

Based on this blog post, a Microsoft colleague told us this:

“I double checked the Windows bug database and they are aware of the problem. No ETA on a fix yet unfortunately. Your workaround is what’s suggested to temporarily get around the error, although it is not suggested as a long-term fix.”

Alternative Solutions

This section was added after our initial workaround and is based on the experience of many users struggling with this problem.

The problem is often caused because the local machine is patched with the Windows Update and the machine it’s connecting to is not patched for the CredSSP issue. If both systems were patched then this error would not occur.

There are two options:

Update the Target Machine

Update the target machine with the patch for the CredSSP issue (preferable).

Update the Local Machine

In many cases, you don’t have the option to modify anything on the target machine. You may even be prevented from modifying your own machine, but assuming you have administrator rights, you can change the Group Policy on your local machine to use the Vulnerable setting.

Big picture, it’s ridiculous to lower one’s security settings to connect to a machine that wasn’t updated. It would be much better if it prompted or automatically connected to lower level machines without turning off the higher security level for everything else. All it takes is one target machine that you can’t modify to force this change on your machine. But at least you can get your work done.

  1. Enter run “gpedit.msc” to edit group policy, or from Windows start, enter “Group Policy” and select “Edit group Policy”:
    1. Windows 10
    2. Windows 7
  2. From the treeview, choose Computer Configuration -> Administrative Templates -> System -> Credentials Delegation
  3. Select “Encryption Oracle Remediation” from the right pane (if it’s not there, it probably means your machine wasn’t patched):
  4. Enable and set the Protection Level to Vulnerable:

Hope this helps.


Additional Problem: Cannot Connect via VPN

We’ve discovered problems with VPN connection if the PC has Remote set to the higher security level.

The network connection fails with error: Cannot load the Remote Access Connection Manager service. Error 711:

Lower Your Remote Desktop Security to have the Security to Make the VPN Connection

Apparently, the Remote Desktop setting on the client side impacts its ability to connect via VPN to the host side.

By lowering the setting to less secure for others to connect to the PC, the PC can now successfully connect to the VPN. What a mess.

Additional Discussions

I’ve also been involved in other online discussions:

Summary

It’s late August, and it’s shocking that this problem remains after so many months. I am extremely frustrated by the Windows update policies and Microsoft’s inadequate testing before these security patches are deployed. This is very disruptive and dangerous to many organizations trying to fulfill their missions expecting their PCs to be reliable.

Microsoft security “purists” claim the current approach is necessary to address the serious threats facing users. I guess it wouldn’t be an issue if the updates worked without disruption. However, the downside of this medicine may exceed the illnesses they are trying to prevent.

Hope you are able to resolve this and move on.


Additional Remote Desktop Connection Resources

Nov 28

Microsoft Access is 25 Years Old!

birthday25Microsoft Access is celebrating its 25th year this month. It’s an amazing accomplishment for a software product to be so successful for so many years. We at FMS were there since the beginning.

Read our first hand, historical account of watching Microsoft Access take over the Windows desktop database market, and how we became the world’s leading 3rd party developer of Microsoft Access products.

Discover how we watched MS Access rise from nothing to the leading Windows desktop database application. This directly caused the implosion of Borland International which previously dominated the desktop database industry. Witnessing this in person was an amazing experience of how quickly technology can change established, large software businesses.

This article was originally published by Microsoft on their website for the 10 Year Anniversary Celebration of Access (October 2002)

The video is from the November 1992 COMDEX conference where a very young Bill Gates personally announced the debut of Microsoft Access. He remains a big fan of Access and was actively involved in its design and development.
bill-gates

Jan 05

Sean Hannity Radio Show Interview on Julian Assange, WikiLeaks, Russian Hacking, and Cyber Warfare

sean-hannity-radioBackground

The day after an amazing personal interview of Julian Assange by Sean Hannity aired on his TV show, FMS President Luke Chung was invited to discuss the related technology on his radio show.

Hannity traveled to London to interview Julian Assange at the Ecuadorian Embassy where he’s seeking asylum. They discussed an overview of Assange’s role as founder of WikiLeaks, and their obtaining and publishing the emails from the Democratic National Committee the weeks before the US Presidential election. Some people attribute Hillary Clinton’s loss to the revelations in those emails especially from John Podesta, the former White House Chief of Staff and Chairman of the Clinton campaign. They are also accusing the Russians for hacking (stealing) and providing the data to Assange so Donald Trump could win the election.

Radio Show

On January 4, 2017, I was on the radio show with Sean Hannity and Brigadier General Eli Ben Meir, former Israeli Military Intelligence chief. The three of us discussed the WikiLeaks disclosures. I commented specifically on:

  • Cyber attacks and the security breach at OPM disclosed non-classified government employees and by omission who were covert at American Embassies globally.
  • Noting Julian Assange’s careful word choices to exclude Russia as source without excluding them as the ultimate source of his sources.
  • The need for WikiLeaks’ to keep their sources confidential and how they amplified the data from Bradley Manning and Edward Snowden.
  • Different approaches to preventing cyber attacks depending on the cause.
    “It’s one thing when someone steals your car because they broke into it. It’s another thing when someone steals your car because you left your keys in the ignition.”

Here’s the audio of the show:

My segment starts at the 6:50 mark. Sean and General Meir speak first, then I start around 9:25. Final comments at 14:15 and it wraps up by 14:50.

Additional Issues

Only a limited amount of information can be discussed in such a short interview. Some additional issues to consider are:

Data Security

Securing data over the internet and inside organizations is very challenging. Threats may come from:

  • External hacks that need to be monitored and defeated
  • Internal people who unintentionally leave the front door unlocked
  • Internal people who intentionally leak information

Different solutions are required for each type of threat. Some are at the software vendor, design, and developer level, while others involve end-user training, background checks, and monitoring.

Applications can be built so that simply disclosing a user name and password doesn’t compromise the whole system by require two-factor authentication and registering devices that can use those credentials.

Unfortunately, many systems were built well before today’s cyber threats existed. The cost of making those systems more secure without breaking their existing functionality will be daunting and expensive. In many cases, the original source code, development environment and/or vendor are long gone, so the only option is to replace them which is also very expensive and time consuming.

Julian Assange and WikiLeaks are Not Heroes

We need to keep in mind that the WikiLeaks exposed top secret US information by publishing the disclosures from Manning and Snowden. Lives were put at risk and lives may have been lost because of those publications. The Arab Spring was inflamed in part by the disclosure of diplomatic communications and one could argue the human tragedy in Syria is tied to this as well. While Republicans are celebrating and defending Assange and WikiLeaks now for the DNC emails, the tables may turn very quickly.

Data That’s Not Exposed May be More Dangerous

While many are focused on the DNC emails, it’s not unreasonable to assume the people who hacked that also got the RNC emails. Data can be power, and in the wrong hands, data can be used for nefarious purposes such as blackmail.

If the RNC data were compromised, we should be extremely worried if the hackers discovered it was more valuable to keep private than public. Whether they use it directly or sell it to another party or country, the information can make victims puppets by threatening the exposure of their personal data. It’s not uncommon during E-discovery of an email server to discover all sorts of inappropriate language, behavior and activity conducted by individuals in an organization. Disclosures of affairs, homosexual activity, underage sex, bribery, unethical business dealings, breeches of confidentiality, collusion, and actual crimes are often found in email threads and can be used for blackmail.

Additional Resources

Jul 01

Microsoft Office365 Exchange Online Service Performance Degradation and SMTP Problems

office365A few years ago, we migrated our email service to Microsoft’s Office365 cloud service. Overall, it’s been very reliable and eliminated the challenges we had hosting Exchange ourselves. It let us get to our emails using Outlook installed on Windows, any internet browser, and smartphones. Office365 also offered other Office product online (Access Web Apps, Excel, Word, etc.), SharePoint and OneDrive Business.

Unfortunately, on the morning of June 30th, we discovered:

  • Delays sending and receiving emails
  • Some emails were bouncing back from recipients who couldn’t validate our Office365 Exchange Server’s SMTP (protection.outlook.com) with our domain name. That meant the Exchange SMTP server was no longer considered a trusted sender of emails from the @fmsinc.com domain.
  • Our use of the Office365 SMTP server to send emails with our Total Access Emailer product was also failing to authenticate against the server

The problems began the evening before. Needless to say, we aren’t happy about this experience which impacted us and our clients using Office365. Reports are that it affects Office365 customers across North America.

exchangeContacting Microsoft, they confirmed problems with the health of their Office365 Exchange Server. Throughout the day, problems lessened but persisted. We hope the problems are resolved soon and that we’ll understand what went wrong once we overcome the immediate crises.

These are the reports we’ve received from Microsoft. We’ll keep you updated as we learn more:


Exchange Online Service Degraded

This is what the Office365 Admin portal shows for Service Health:

Office365Issues

Office365Health


EX71628 – E-Mail and calendar access – Restoring Service

Jun 29, 2016 12:11 PM

CURRENT STATUS

Our investigation determined that an existing transport feature which is designed to expedite the delivery of email messages became degraded, which caused impact to email delivery for a subset of users. We’re bypassing the affected feature to restore service

User Impact

Users may be unable to send email messages through the Exchange Online service. Email messages may appear to be stuck in the Drafts or Outbox folders.

Scope of Impact

A few customers have reported this issue, and our analysis indicates that for most customers, it’s unlikely that many users would report impact related to this event.

  • Start Time: Thursday, June 23, 2016, at 3:00 PM UTC

Preliminary Root Cause

An existing transport feature that is designed to expedite the delivery of email messages became degraded, which caused impact to email delivery for a subset of users

EX71628 – E-Mail and calendar access – Extended recovery

Jun 30, 2016 2:18 PM

Current Status

We’ve developed an additional fix to address the underlying cause of the issue. We’re preparing to deploy the fix to the affected environment to ensure that the issue does not reoccur.

User Impact

Users may be unable to send email messages through the Exchange Online service. Email messages may appear to be stuck in the Drafts or Outbox folders.

Scope of Impact

A few customers have reported this issue, and our analysis indicates that for most customers, it’s unlikely that many users would report impact related to this event.

  • Start Time: Thursday, June 23, 2016, at 3:00 PM UTC

Preliminary Root Cause

An existing transport feature that is designed to expedite the delivery of email messages became degraded, which caused impact to email delivery for a subset of users.

Next Update by: Saturday, July 2, 2016, at 7:00 PM UTC


EX71674 – E-Mail timely delivery – Service restored

Jun 30, 2016 7:35 PM

Final Status

We’ve confirmed that the remaining message queues have now drained after implementing a configuration change to optimize message filtering.

User Impact

Users were experiencing delays when sending and receiving email messages. Affected users may have received Non-Delivery Reports (NDR) when sending email messages.

Scope of Impact

Customer reports indicated that many users likely experienced impact related to this event. Our analysis indicates that this issue may potentially have affected any of your users attempting to send or receive mail.

  • Start Time: Thursday, June 30, 2016, at 2:30 PM UTC
  • End Time: Thursday, June 30, 2016, at 11:30 PM UTC

Preliminary Root Cause

The infrastructure responsible for processing Exchange Online Protection (EOP) message filtering became degraded.

Next Steps

  • We’re analyzing performance data and trends on the affected systems to help prevent this problem from happening again.
  • We’re reviewing our code for optimizations and automated recovery options.
  • We’ll publish a post-incident report within five business days.

EX71674 – E-Mail timely delivery – Service restored

Jul 1, 2016 12:08 AM

Final Status

We’ve rolled out the fix and confirmed that service is restored. Any meeting requests created during the outage will need to have the conference room calendar removed and readded to book the room.

User Impact

Users that attempted to create a meeting request with a conference room calendar were unable to successfully book a conference room. This lead to conference rooms being booked by multiple resources.

Scope of Impact

A few customers reported this issue, and our analysis indicated that this may have affected any users attempting to use this feature.
  • Start Time: Monday, June 27, 2016, at 6:00 PM UTC
  • End Time: Friday, July 1, 2016, at 2:54 AM UTC

Preliminary Root Cause

A recent update affected the ability for calendar invite requests to successfully book conference rooms.

Next Steps

  • We’re reviewing our deployment and provisioning procedures to help prevent this kind of problem in the future.
  • We’ll publish a post-incident report within five business days.