MS3 | Cloud Integration Solutions

Let’s say you’re looking for something in an image — e.g. a tumor in a CT scan.  Most likely, you’ll train a deep neural network (or “DNN”) to both flag the scan as containing a tumor, and to highlight the region where the tumor is found.  The neural net is able to take in millions of pixels as input, pool them into groups based on proximity, and compare those grouped pixels against learned patterns.  With each layer of the net, you recognize higher-level patterns until you finally get to a target class (“Tumor” = True). Neural networks are great for high-dimensionality pattern-recognition where you have a lot of training examples to train a LOT of parameters.  So it’s typically the right choice for image recognition.

That doesn’t work so well for presumably relatively low-dimension, low-volume questionnaire data Verily will be using in their just-launched (Santa Clara and San Mateo counties only, at present) COVID-19 risk screening and testing tool.

They’ll most likely use a decision tree.

Decision trees are pretty intuitive.  In the simplest case, you just run through the list of known symptoms of this particular disease, and if you have enough of them, then there’s some probability you have the disease and should be screened.  Such a decision tree can be manually constructed by subject matter experts, even without any training data.

But sometimes the answer is not so obvious.  E.g. You recently returned from Italy, and you are congested and short of breath, but you also have a heart condition and seasonal allergies.  Should you be screened, given the scarcity of available test kits?

Manually-defined decision trees have a much harder time managing complexity.  So in this case, we should train the decision tree via supervised learning, using past questionnaires from patients who tested positive and patients who tested negative.

The tree “learns” how to branch the decision tree by computing the “information gain” at each branch.  Ideally, when we branch the decision tree, we want all the positive diagnoses to go one way and all the negative diagnoses to go the other way.  That won’t happen, of course, but we might get closer to that ideal, where two-thirds of the positives go one way and one-third goes the other way.  That’s considered information gain. And all we do is keep computing which attribute, on which split, gives us the highest information gain, all the way down the tree.

Typically, we’ll stop at some max-depth, because we’ll overfit the model otherwise.  I.e. We’ll get a decision tree that’s too specifically-tuned to the training data and can’t transfer to new data.

This trained decision tree now reflects the best information available to us in the sample data.  Sort of. When you randomly selected a portion of the sample data to use for training the decision tree, you left out about 20% of your data for testing.  It would be a good idea to resample your training data and re-train the decision tree. Probably several times.

At this point, you have several trees, which will differ in non-trivial ways.  And each tree returns a result, which acts as a vote. And the plurality now decides whether you should get screened, and also provides a level of confidence in that decision.

The good news is that training this ensemble of decision trees (or forest, if you will) isn’t that difficult or computationally complex.  The real challenge for a tool like this is ensuring the data is accurate, both when training the tool, and when it is in production. Thus, there are 3 primary design challenges for a company like Verily (subsidiary of Alphabet and sibling to Google) in building this tool:

  1. Gathering and extracting initial diagnostic data from existing health records
  2. Selecting the features to use in training the decision trees and then to incorporate into the questionnaire
  3. Structuring the diagnostic tool’s data intake wording and question-sequence to maximize accuracy of self-reported data.  

Imagine you’re in a COVID-19 hotspot, and you have a sore throat and fever.  You’re already convinced you need to be screened, so you may be inclined to answer every question to maximize the likelihood you’ll get the recommendation for screening.  Or perhaps you’re the opposite, and you’ll downplay and minimize symptoms to avoid being screened.

Big Idea: It’s likely Verily will use a decision forest to recommend screenings to users.  However, the design challenges of this application, as well as those of any successful AI application, extend well beyond the training of the classification model.  

I look forward to watching the roll-out of this tool from my Minnesota bunker constructed entirely from 2-ply, extra soft toilet paper.

About the Author: Eric Nelson

TEAM LEAD · SENIOR SOFTWARE ENGINEER

Eric Nelson is a Senior Software Engineer with MS³, Inc., living in Minneapolis, Minnesota with his wife Alisa, 7-year-old daughter Freyda, and 5-year-old son Arthur. Eric received his Bachelor of Science in Electrical Engineering from University of Minnesota and worked in photovoltaics and thin films. In 2015, he founded his own cloud software consulting firm where he trained dozens of student interns in software design and development skills. He is now focused on building large-scale, secure, futureproof production AI applications and software systems for smart brands as a member of the MS³ family.


Stories from the Trenches: Part 1 of 3

We recently finished an engagement with a Fortune 500 firm where we had some freedom to decide where to initiate a Data Science project, based on highest value to the business.  Naturally, improving the success rate of a profit center — e.g. conversion rates between each stage in the sales pipeline — is the fastest way to see obvious revenue growth from a new project.  The internal analytics group pointed us to a particular stage that was clearly struggling, so we dove in.

In this industry, every solution is highly customized, so when a company sends a prospect a quote, they’ve already invested significant time and money into the deal.  Many requirements need to be uncovered and many product configurations need to be considered to meet those requirements. You wouldn’t expect a company to incur the substantial effort and expense to generate a custom quote unless there was a good chance to win the contract.  In contrast, this company was seeing 17% conversion from the Quote stage. So, after all that time and effort gathering requirements and architecting a solution, only 1 in 6 contracts quoted proceed to the next stage in the sales pipeline. That’s a problem.

Always distrust first impressions (though they may be justified in the end)

There are a lot of factors that could contribute to this low conversion rate: poor lead qualification, long quote turnaround times, underperforming sales personnel, poor pricing models, etc.  Or maybe incurring some expense is part of a sales tactic to induce reciprocity by the prospect. A good scientist suspends judgment until there is a clear body of evidence to support a hypothesis.  So we kept digging into historical data, and we found 2 critical insights: 1) 20% of deals that did convert from quote to the next stage reverted back to get re-quoted due to missed requirements, 2) a disproportionate number of exit surveys from former customers (i.e. churned customers) pointed to misalignment between requirements and solutions.

In other words, not only were 5 out of 6 quoted prospects not interested in the quoted solutions — a surprising number who were initially interested were ultimately dissatisfied with the solution.  With insights in-hand, we set off to find a solution architect who could help us understand the current quoting process, in all its gory detail.  

In the immortal words of Sir Arthur Conan Doyle…

Down the rabbit hole

Our first meeting with the lead solution architect revealed a very complex set of product configuration rules guiding the solutioning process, which he admitted was challenging for new solution engineering personnel to learn.  They had built tools in the past (in Excel) to assist in computing certain mathematical relationships using manually-defined rule systems. However, those tools soon became outdated as products changed, and they were painful to maintain. Ultimately, they had to rely primarily on experience to guide their solutioning practices.  Naturally, the newer personnel made a lot of mistakes. Some embarrassingly obvious. And due to time constraints, the senior staff aren’t able to check every solution thoroughly before it goes out the door. Clearly, the quoting process requires an intervention. But what intervention will most reliably and economically deliver the sought-after performance improvement?

Coming up with roses

It is reasonable at this point in a Discovery to step back and look at the big picture of the organization.  Where is the company headed? Are major product changes or infrastructure overhauls planned within the next 5 years?  How does the company want to position itself to its customers/prospects? Is it a technology leader, building next-gen services and support tech?  Is it a technology follower, adopting only well-proven technologies as part of the Late Majority?

Talking with executives, we learned the major competitors were eating our client’s lunch, in part because of more effective technology infrastructure supporting more valuable service delivery to customers.  This company was hungry to lay a foundation for internal technological innovation — which was why we were there in the first place. So we were encouraged to begin laying that foundation with our proposed solution.  There were 3 clear options to select from:

  1. The most obvious solution to human performance gaps is better training for the new guys.  And that would likely be a valid solution here, especially if coupled with a customized Learning Management System that effectively organized the curricular content, labs, and assessments to better scale the senior personnel’s available time.  The upfront investment by senior resources would be substantial, and there would be ongoing maintenance to keep the content up-to-date. Not the most innovative choice.
  2. Another obvious solution is better quality control for quotes sent to the customer.  This would also rely heavily on senior resources, who would be checking the more junior resources’ work, but it’s possible the senior resources’ actions could be monitored and used to train an AI to detect common mistakes.  The primary innovation here would be a collaborative, versioned configuration management system that “listens in” on changes made by senior resources and learns by observation.
  3. Finally, we can revisit the concept of creating tools to actually recommend solution components based on requirements — i.e. requirements go in one side, and solution recommendations come out the other.  This dramatically reduces time and complexity of the quoting operation, as the human operator faces a much smaller total body of decisions. This solution would rely the least on senior personnel. It would rely instead on historical requirements documentation and their corresponding solutions, which together would serve as the data and labels for a supervised learning-based recommendation engine.  This option has a distinct advantage in that it dramatically reduces the time and cost to quote a solution by minimizing human decision-making to only those critical edge-conditions. It also makes best use of a largely untapped asset: historical data.

The 3rd solution represents a step-change in the company’s business: Faster, more reliable, more economical quotes through a technological innovation.  It’s an inspiring vision, which aligns well with the company’s eagerness to prove itself as a technology leader. We went for it. First step was to assess the quality of the existing requirements documentation — which we will dive into in Part 2: Turning “Requirments Documentation” Into Gold.

About the Author: Eric Nelson

TEAM LEAD · SENIOR SOFTWARE ENGINEER

Eric Nelson is a Senior Software Engineer with MS³, Inc., living in Minneapolis, Minnesota with his wife Alisa, 7-year-old daughter Freyda, and 5-year-old son Arthur. Eric received his Bachelor of Science in Electrical Engineering from University of Minnesota and worked in photovoltaics and thin films. In 2015, he founded his own cloud software consulting firm where he trained dozens of student interns in software design and development skills. He is now focused on building large-scale, secure, futureproof production AI applications and software systems for smart brands as a member of the MS³ family.

Filed Under: IoT

Written By: Eric Nelson

This is the 1st post in a series on “Optimizing Commercial Building UX, Asset Health, and Energy Conservation with IoT and AI,” leading up to
my talk on April 25th at IoTFuse2019.

 

How Much Energy is Wasted in my Portfolio per Year?

What is the Primary Source of Waste?

How Much Money Could I Save by Optimizing Energy
Spend without Sacrificing Tenant Comfort?

These are a couple questions which can be asked when managing or assessing operational costs of a commercial property. Billions of $USD are spent each year operating commercial office buildings. Most are still operated based on simple automation rules, with more complex behavior relying on manual procedures. There are, however, pioneering companies who are building networks of sensors and pattern-learning software to monitor and automate sophisticated building behavior which saves costs while improving the tenant experience.

What is the Typical Energy Cost Profile for Commercial Office Buildings in the U.S.?

The U.S. Energy Information Administration (EIA) does a periodic survey of energy consumption in commercial buildings (called CBECS) in all 50 states. In 2015, they released the results of the 2012 survey. There’s another survey currently underway for CY 2018, but, clearly, it’ll be a while until those results are released.
**Note: All estimated qualities are extrapolated from a representative sample size of ~6,700 surveyed buildings. Read more about CBECS Methodology & Research

In 2012, there were over 1 million U.S. commercial office buildings, enclosing nearly 16 billion square feet, in operation for an average of 55 hours per week. Annually, total energy delivered to those building costs around $31 billion. That’s electricity ($26B), natural gas ($2.3B), district heat ($1.8B), and fuel oil (0.5B). “District heat” is hot water or steam piped out to buildings from a central heat generation plant. It tends to be more prominent in the Mid- and South-Atlantic regions. And ancient Rome.

Electricity expenditures obviously dwarf those of the other energy sources. So where does all that energy go? Almost 25% goes to Ventilation, ~20% to Computing, ~17% to Lighting, ~13% to Cooling, and the remaining 25% goes to other office equipment, refrigeration, space heating, and, of course, the eponymous Other stuff that isn’t tracked. Lighting used to be a bigger consumer, but more CFLs replaced a lot of incandescent bulbs over the last 10 years and many conventional F40 fluorescent tubes have been replaced by models up to 20% more efficient.

We don’t have more detailed data on the top energy consumers (Ventilation & Computing, totaling 45%), but we do have Lighting and Cooling data, totaling 30% — or about $8 Billion annual electricity spend. Here’s some interesting usage information:

  1. Nearly 40% of all commercial office buildings are 100% lit when the building is on.
  2. Over 60% of all buildings cool 100% of the space when the building is on.

That includes vacant offices and conference rooms, as well as mostly-vacant large open areas. We don’t have the data, but we can presume the numbers for cooling are probably similar for ventilation. I.e. Perfectly good air is being vented out of vacant rooms.

The majority of remaining buildings are lit and cooled 50-99%. It’s safe to presume there are some inefficiencies in these buildings as well.

The Problem with Current Control Technology is that it’s Difficult to Make Buildings More Efficient without Making them Less Comfortable.

The typical Building Management System (BMS) operates on relatively simple rules (turn on at this time and turn off at this time) to control relatively large sections of the building. Oftentimes, the only way to ensure tenant comfort is to turn everything on well in advance of working hours.

So that’s the control problem. But there’s also a logic problem. It’s difficult to specify rules for how human tenants will use a commercial office building. A room left vacant one day may be occupied the next day. More bodies may be in the building on Wednesday than Friday. Conducting a survey of when people work, and which rooms they use most frequently would give temporary insight, but we could never specify such detailed rules into a Building Automation System. The rules would change too frequently, and inevitably, tenant comfort would suffer as rule changes lagged behind behavior.

We’d prefer to take appropriate action based on real-time, measured information. E.g. How many bodies are in the building right now, radiating heat and exhaling CO2 to vent? What is current room temperature and CO2 level? What is the ambient temperature outside? Is anyone using Conference Room A?

Actually, because it takes time to cool & vent spaces, we’d really like to take action based on an accurate prediction of the near-future. E.g. Given it’s a Thursday, how many bodies will be in the 3rd floor atrium in 30 minutes? Who will be using Conference Room A in 20 minutes and what temperature setpoint do they like?

So, we measure the building data at high detail, and the we learn rules for managing it at a similar level of detail. Logic and control.

How Does an IoT and AI Solution Reduce the Building Energy Cost Profile?

Building data comes in through a fairly dense swarm of sensors, which can be installed, for example, in the ceiling, on the same power line and data bus as the lighting system. The sensors measure occupancy, temperature, CO2 levels, and lighting output. Sensor clusters are then mapped to XY coordinates in a floor plan. Conference rooms and other enclosed spaces will have their own sensor clusters, and so on. This network of sensors forms a “nervous system”, informing a building manager of current status at high detail in real-time.

Other sensors measure the rate of volumetric air being moved in and out of each space, the amount of energy is expended to cool that air to a temperature setpoint, and which chillers are consuming what proportions of that energy. Temperature and CO2 readings near each vent will be strongly correlated to its airflow readings. There is a math model for how changes to airflow will change nearby temperature and CO2 levels.

With sensors in place, we can run experiments to learn a unique model for each vent. Naturally, spaces near doors and other high human throughput zones will have different relationships than relatively unperturbed spaces. And the relationships will change over time as human behaviors change and asset performance degrades. Thus, models and rules will need to be continuously re-learned. Each relationship, however, gives us high-resolution levers to pull to get the comfort outcomes our tenants want. All while turning on as little of the building as possible, minimizing costs.

The aim is for no building to ever be 100% cooled, 100% vented, or 100% lit, unless it is 100% occupied. Our next step to that end is to define a strategy for initiating deployment of IoT and AI in commercial office buildings, both in new construction and retrofitting existing stock.

But Does this Technology Really Work in Practice?

There is a growing number of success stories in implementing IoT and AI in commercial office buildings. Here are a few.

Rudin Management Company collaborated with Columbia University back in 2010 to build Di-BOSS (Digital Building Operating System) in order to read data from, and remotely control, all the different equipment in their commercial buildings. Eventually Intel came in and migrated the database from SQL to NoSQL and from prem to cloud, in order to scale the solution beyond a single building. A year later, it was running 17 buildings. And according to Intel, energy use dropped by 9% and costs went down nearly $1M in the first year in just one building. Not only that, but hot/cold dropped by 70%. The OS, now called Nantum, “forecasts and proactively predicts optimal settings for HVAC and other systems”, and Rudin sees annual savings of 30% energy consumption and over $5 million cost saving across the 17 buildings.

In 2014, Deloitte Netherlands, in collaboration with OVG Real Estate, built what would come to be known as “the greenest, most intelligent building in the world.” Known as The Edge in Amsterdam, the new construction was rated the world’s most sustainable building by BREEAM, scoring 98.36% across 10 sustainability metrics (the highest ever awarded at the time). A primary strength of the building design is a swarm of 28,000 sensors monitoring lighting, air quality, temperature, and occupancy. This real-time data is fed into a custom monitoring and usage pattern-learning application developed by Deloitte. According to Deloitte, “Long term emerging patterns showing light use of certain locales on certain days can lead to rooms or even entire floors being closed off to save energy.” The resulting energy profile can be sustained almost entirely by 5900 m2 of rooftop solar panels.

Energy saving, of course, is only the beginning of the benefits from real-time monitoring of commercial office buildings. For example, when Teradyne implemented Watson IoT and Tririga from IBM, they included sensors from monitoring contaminate levels in the heating and cooling water loop. IBM also cites instances of forecasting end-of-life for large assets (like air handlers), enabling better financial management and less down time for business maintenance. The Edge in Amsterdam credits hot-desking (where workspaces are booking on-demand vs permanently allocated – because they’re often vacant) with increasing effective capability by more than 140%.

It Sounds Good. How Would I Get Started?

The strategy for implementing IoT and AI in commercial office buildings has roughly 7 parts:

  1. Define the objectives for the technology, based on an anticipated ROI, and the intended implementation schedule.
  2. Design the network of sensors and IoT gateways to embed into the building.
  3. Collect real-time sensor data into both structured and unstructured data stores.
  4. Model relationships between inputs and outputs using regression and other machine learning techniques and then deploying those models for on-demand interface.
  5. Expose both raw sensor data and inferences to user-facing applications via API.
  6. Design easy-to-use, high-productivity user interfaces for building managers and occupants.
  7. Continuous improvement of inference models using reinforcement learning.

AND that’s where we’re heading next!

Get notified about the next article in the “Optimizing Commercial Building UX, Asset Health, and Energy Conservation with IoT and AI” by following us on Twitter (@ms3-inc)

About the Author

Eric Nelson is a Sr. Software Engineer with MS3, Inc., living in Minneapolis, MN with his wife Alisa, 6yo daughter Freyda, and 4yo son Arthur. Eric received his BS in Electrical Engineering from University of Minnesota and worked in photovoltaics & thin films, both at MN tech start-ups. In 2013, Eric helped found an inner-city charter middle school and taught courses in technology and entrepreneurship. In 2015, he founded his own cloud software consulting firm Augment LLC. And began training student cohorts in software design & development skills. He is now focused on building large-scale, secure, futureproof production software systems for smart brands as a member of the MS3 family. Meanwhile, he continues his effort to develop a scalable programmer training network for high school and college students. He also serves on the board of Minnesota Innovates, a nonprofit dedicated to cultivating MN technology startups building products in emergent technologies (e.g. AI, AR/VR, Gene-editing, Drones, etc.)

 

About

Careers

Partners

Solutions

Contact

From the Blog

ms3-mountain
white-mountain