Hitting the Books: The women who made ENIAC more than a weapon

After Mary Sears and her team had revolutionized the field of oceanography, but before Katherine G. Johnson, Dorothy Vaughan and Mary Jackson helped put John Glenn into orbit, a cadre of women programmers working for the US government faced an impossible task: train ENIAC, the world’s first modern computer, to do more than quickly calculate artillery trajectories. Though successful — and without the aid of a guide or manual no less — their names and deeds were lost to the annals of history, until author Kathy Kleiman, through a Herculean research effort of her own, brought their stories to light in Proving Ground: The Untold Story of the Six Women Who Programmed the World’s First Modern Computer.

Proving Grounds Cover
Grand Central Publishing

Excerpted from the book Proving Ground: The Untold Story of the Six Women Who Programmed the World’s First Modern Computer by Kathy Kleiman. Copyright © 2022 by First Byte Productions, LLC. Reprinted with permission of Grand Central Publishing. All rights reserved.


Demonstration Day, February 15, 1946

The Moore School stood ready as people began to arrive by train and trolley. John and Pres, as well as the engineers and deans and professors of the university, wore their best suits and Army officers were in dress uniform with their medals gleaming. The six women wore their best professional skirt suits and dresses.

Kay and Fran manned the front door of the Moore School. As the scientists and technologists arrived, some from as far as Boston, the two women welcomed them warmly. They asked everyone to hang up their heavy winter coats on the portable coat racks that Moore School staff had left nearby. Then they directed them down the hall and around the corner to the ENIAC room.

Just before 11:00 a.m., Fran and Kay ran back to be in the ENIAC room when the demonstration began.

As they slid into the back of the room, everything was at the ready. At the front of the great ENIAC U, there was space for some speakers, a few rows of chairs, and plenty of standing room for invited guests and ENIAC team members. Across the room, Marlyn, Betty, and Jean stood in the back and the women smiled to each other. Their big moment was about to begin. Ruth stayed outside, pointing late arrivals in the right direction.

The room was packed and was filled with an air of anticipation and wonder as people saw ENIAC for the first time.

Demonstration Day started with a few introductions. Major General Barnes started with the BRL officers and Moore School deans and then presented John and Pres as the co-inventors. Then Arthur came to the front of the room and introduced himself as the master of ceremonies for the ENIAC events. He would run five programs, all using the remote control box he held in his hand.

The first program was an addition. Arthur hit one of the but-tons and the ENIAC whirled to life. Then he ran a multiplication. His expert audience knew that ENIAC was calculating it many times faster than any other machine in the world. Then he ran the table of squares and cubes, and then sines and cosines. So far, Demonstration Day was the same as the one two weeks earlier, and for this sophisticated audience, the presentation was pretty boring.

But Arthur was just getting started and the drama was about to begin. He told them that now he would run a ballistics trajectory three times on ENIAC.

He pushed the button and ran it once. The trajectory “ran beautifully,” Betty remembered. Then Arthur ran it again, a version of the trajectory without the punched cards printing, and it ran much faster. Punched cards actually slowed things down a little bit.

Then Arthur pointed everyone to the grids of tiny lights at the top of the accumulators and urged his attendees to look closely at them in the moments to come. He nodded to Pres, who stood against the wall, and suddenly Pres turned off the lights. In the black room, only a few small status lights were lit on the units of ENIAC. Everything else was in darkness.

With a click of the button, Arthur brought the ENIAC to life. For a dazzling twenty seconds, the ENIAC lit up. Those watching the accumulators closely saw the 100 tiny lights twinkle as they moved in a flash, first going up as the missile ascended to the sky, and then going down as it sped back to earth, the lights forever changing and twinkling. Those twenty seconds seemed at once an eternity and instantaneous.

Then the ENIAC finished, and darkness filled the room again. Arthur and Pres waited a moment, and then Pres turned on the lights and Arthur announced dramatically that ENIAC had just completed a trajectory faster than it would take a missile to leave the muzzle of artillery and hit its target. “Everybody gasped.”

Less than twenty seconds. This audience of scientists, technologists, engineers, and mathematicians knew how many hours it took to calculate a differential calculus equation by hand. They knew that ENIAC had calculated the work of a week in fewer than two dozen seconds. They knew the world had changed.

Climax complete, everyone in the room was beaming. The Army officers knew their risk had paid off. The ENIAC engineers knew their hardware was a success. The Moore School deans knew they no longer had to be worried about being embarrassed. And the ENIAC Programmers knew that their trajectory had worked perfectly. Years of work, effort, ingenuity, and creativity had come together in twenty seconds of pure innovation.

Some would later call this moment the birth of the “Electronic Computing Revolution.” Others would soon call it the birth of the Information Age. After those precious twenty seconds, no one would give a second look to the great Mark I electromechanical computer or the differential analyzer. After Demonstration Day, the country was on a clear path to general- purpose, programmable, all- electronic computing. There was no other direction. There was no other future. John, Pres, Herman, and some of the engineers fielded questions from the guests, and then the formal session finished. But no one wanted to leave. Attendees surrounded John and Pres, Arthur and Harold.

The women circulated. They had taken turns running punched cards through the tabulator and had stacks of trajectory printouts to share. They divided up the sheets and moved around the room to hand them out. Attendees were happy to receive a trajectory, a souvenir of the great moment they had just witnessed.

But no attendee congratulated the women. Because no guest knew what they had done. In the midst of the announcements and the introductions of Army officers, Moore School deans, and ENIAC inventors, the Programmers had been left out. “None of us girls were ever introduced as any part of it” that day, Kay noted later.

Since no one had thought to name the six young women who programmed the ballistics trajectory, the audience did not know of their work: thousands of hours spent learning the units of ENIAC, studying its “direct programming” method, breaking down the ballistics trajectory into discrete steps, writing the detailed pedaling sheets for the trajectory program, setting up their program on ENIAC, and learning ENIAC “down to a vacuum tube.” Later, Jean said, they “did receive a lot of compliments” from the ENIAC team, but at that moment they were unknown to the guests in the room.

And at that moment, it did not matter. They cared about the success of ENIAC and their team, and they knew they had played a role, a critical role, in the success of the day. This was a day that would go down in history, and they had been there and played an invaluable part.

GM is using its Ultium battery tech for a lot more than EVs

I wasn’t kidding when I told you that GM is going all-in on Ultium, the battery technology behind the company’s electrification efforts, not to mention an entire generation of Chevy and GMC EVs. On Tuesday, the automaker announced that it is expanding its portfolio into energy management services — think big stationary batteries to store rooftop-generated solar power on a home or business — with its new spin-off business, GM Energy.

The new venture will be comprised of three smaller ones: Ultium Home, Ultium Commercial and Ultium Charge 360, offering “solutions ranging from bi-directional charging, vehicle-to home (V2H) and vehicle-to-grid (V2G) applications, to stationary storage, solar products, software applications, cloud management tools, microgrid solutions, hydrogen fuel cells and more,” according to GM’s announcement on Tuesday. 

The new company will be partnering with a number of established firms and utilities in the energy industry. For example, GM will be working with SunPower to develop and market a integrated home energy storage system that incorporates an electric vehicle with solar panels and battery banks to enable easy Vehicle-to-Home (V2H) power transfers. GM plans to have that home energy system ready for sale alongside the release of the EV Silverado next fall, 2023

Additionally, GM Energy has partnered with California’s Pacific Gas and Electric utility for another V2H pilot program that will let you run your household appliances off of your EV’s battery during blackouts. Eventually, the company plans to add V2G (Vehicle-to-Grid) capabilities, which will allow you to sell excess energy produced by the solar panels back to your local utility. 

For businesses, Ultium Commercial may help ease the transition to an electrified fleet. Many such existing GM customers, “have fleets of vehicles are looking to electrify their fleets, but aren’t really aware of how to set up the charging infrastructure, how to manage their energy,” Mark Bole, vice president and Head of V2X Battery Solutions at GM said during an embargoed press briefing last week. “And so, not only do we come in as a hardware and software provider, but in a sense, really, as a strategic advisor for these commercial customers.”

“There are more power failures in the US than any other country in the industrialized world,” Travis Hester, vice president of GM EV Growth Operations, added. “There were 25,000 blackouts in California alone last year, over 15 and a half billion dollars of lost commerce, just in California. So when you look at the numbers, there is a desire — and we’re seeing it very clearly from commercial customers reaching out to us and asking us for assistance to deal with some of these problems.”

GM is also transferring its public charging station network, Ultium Charge 360, over to GM Energy. Charge 360 launched in 2021 in Washington, Florida and California. GM partnered with Blink Charging, ChargePoint, EV Connect, EVgo, FLO, Greenlots and SemaConnect to streamline their collective 60,000-plug network of 350 kW Level 3 DC fast chargers and provide “more seamless access” to drivers. The automaker built upon that network this past July, announcing a 500-station “coast-to-coast” expansion in partnership with EVGo. In all, GM hopes to have 2,700 such EV fast charging stations across the US and Canada under its Ultium Charge 360 banner by 2025. 

Hitting the Books: Steve Jobs’ iPhone obsession led to Apple’s silicon revolution

The fates of Apple and Taiwanese semiconductor manufacturer TSCM have grown inextricably intertwined since the advent of the iPhone. As each subsequent generation of iPhone hurtled past the technological capabilities of its predecessor, the processors that powered them grew increasingly complex and specialized — to the point that, today, TSCM has become the only chip fab on the planet with the requisite tools and know-how to actually build them. In his new book, Chip War: The Fight for the World’s Most Critical Technology, economic historian Chris Miller examines the rise of processor production as an economically crucial commodity, the national security implications those global supply chains might pose to America.

Chip War Cover
Simon & Schuster

Excerpted from Chip War: The Fight for the World’s Most Critical Technology by Chris Miller. Reprinted with permission from Scribner. Copyright 2022.


Apple Silicon

The greatest beneficiary of the rise of foundries like TSMC was a company that most people don’t even realize designs chips: Apple. The company Steve Jobs built has always specialized in hardware, however, so it’s no surprise that Apple’s desire to perfect its devices includes controlling the silicon inside. Since his earliest days at Apple, Steve Jobs had thought deeply about the relationship between software and hardware. In 1980, when his hair nearly reached his shoulders and his mustache covered his upper lip, Jobs gave a lecture that asked, “What is software?” 

“The only thing I can think of,” he answered, “is software is something that is changing too rapidly, or you don’t exactly know what you want yet, or you didn’t have time to get it into hardware.” 

Jobs didn’t have time to get all his ideas into the hardware of the first-generation iPhone, which used Apple’s own iOS operating system but outsourced design and production of its chips to Samsung. The revolutionary new phone had many other chips, too: an Intel memory chip, an audio processor designed by Wolfson, a modem to connect with the cell network produced by Germany’s Infineon, a Bluetooth chip designed by CSR, and a signal amplifier from Skyworks, among others. All were designed by other companies.

As Jobs introduced new versions of the iPhone, he began etching his vision for the smartphone into Apple’s own silicon chips. A year after launching the iPhone, Apple bought a small Silicon Valley chip design firm called PA Semi that had expertise in energy-efficient processing. Soon Apple began hiring some of the industry’s best chip designers. Two years later, the company announced it had designed its own application processor, the A4, which it used in the new iPad and the iPhone 4. Designing chips as complex as the processors that run smartphones is expensive, which is why most low- and midrange smartphone companies buy off-the-shelf chips from companies like Qualcomm. However, Apple has invested heavily in R&D and chip design facilities in Bavaria and Israel as well as Silicon Valley, where engineers design its newest chips. Now Apple not only designs the main processors for most of its devices but also ancillary chips that run accessories like AirPods. This investment in specialized silicon explains why Apple’s products work so smoothly. Within four years of the iPhone’s launch, Apple was making over 60 percent of all the world’s profits from smartphone sales, crushing rivals like Nokia and BlackBerry and leaving East Asian smartphone makers to compete in the low-margin market for cheap phones. 

Like Qualcomm and the other chip firms that powered the mobile revolution, even though Apple designs ever more silicon, it doesn’t build any of these chips. Apple is well known for outsourcing assembly of its phones, tablets, and other devices to several hundred thousand assembly line workers in China, who are responsible for screwing and gluing tiny pieces together. China’s ecosystem of assembly facilities is the world’s best place to build electronic devices. Taiwanese companies, like Foxconn and Wistron, that run these facilities for Apple in China are uniquely capable of churning out phones, PCs, and other electronic. Though the electronics assembly facilities in Chinese cities like Dongguan and Zhengzhou are the world’s most efficient, however, they aren’t irreplaceable. The world still has several hundred million subsistence farmers who’d happily fasten components into an iPhone for a dollar an hour. Foxconn assembles most of its Apple products in China, but it builds some in Vietnam and India, too. 

Unlike assembly line workers, the chips inside smartphones are very difficult to replace. As transistors have shrunk, they’ve become ever harder to fabricate. The number of semiconductor companies that can build leading-edge chips has dwindled. By 2010, at the time Apple launched its first chip, there were just a handful of cutting-edge foundries: Taiwan’s TSMC, South Korea’s Samsung, and — perhaps — GlobalFoundries, depending on whether it could succeed in winning market share. Intel, still the world’s leader at shrinking transistors, remained focused on building its own chips for PCs and servers rather than processors for other companies’ phones. Chinese foundries like SMIC were trying to catch up but remained years behind. 

Because of this, the smartphone supply chain looks very different from the one associated with PCs. Smartphones and PCs are both assembled largely in China with high-value components mostly designed in the U.S., Europe, Japan, or Korea. For PCs, most processors come from Intel and are produced at one of the company’s fabs in the U.S., Ireland, or Israel. Smartphones are different. They’re stuffed full of chips, not only the main processor (which Apple designs itself), but modem and radio-frequency chips for connecting with cellular networks, chips for WiFi and Bluetooth connections, an image sensor for the camera, at least two memory chips, chips that sense motion (so your phone knows when you turn it horizontal), as well as semiconductors that manage the battery, the audio, and wireless charging. These chips make up most of the bill of materials needed to build a smartphone. 

As semiconductor fabrication capacity migrated to Taiwan and South Korea, so too did the ability to produce many of these chips. Application processors, the electronic brain inside each smartphone, are mostly produced in Taiwan and South Korea before being sent to China for final assembly inside a phone’s plastic case and glass screen. Apple’s iPhone processors are fabricated exclusively in Taiwan. Today, no company besides TSMC has the skill or the production capacity to build the chips Apple needs. So the text etched onto the back of each iPhone — “Designed by Apple in California. Assembled in China”—is highly misleading. The iPhone’s most irreplaceable components are indeed designed in California and assembled in China. But they can only be made in Taiwan.

Boston Dynamics and other industry heavyweights pledge not to build war robots

The days of Spot being leveraged as a weapons platform and training alongside special forces operators are already coming to an end; Atlas as a back-flipping soldier of fortune will never come to pass. Their maker, Boston Dynamics, along with five other industry leaders announced on Thursday that they will not pursue, or allow, the weaponization of their robots, according to a non-binding, open letter they all signed.

Agility Robotics, ANYbotics, Clearpath Robotics, Open Robotics and Unitree Robotics all joined Boston Dynamics in the agreement. “We believe that adding weapons to robots that are remotely or autonomously operated, widely available to the public, and capable of navigating to previously inaccessible locations where people live and work, raises new risks of harm and serious ethical issues,” the group wrote. “Weaponized applications of these newly-capable robots will also harm public trust in the technology in ways that damage the tremendous benefits they will bring to society.” 

The group cites “the increasing public concern in recent months caused by a small number of people who have visibly publicized their makeshift efforts to weaponize commercially available robots,” such as the armed Spot from Ghost Robotics, or the Dallas PD’s use of an EOD bomb disposal robot as an IED as to why they felt the need to take this stand. 

To that end, the industry group pledges to “not weaponize our advanced-mobility general-purpose robots or the software we develop that enables advanced robotics and we will not support others to do so.” Nor will they allow their customers to subsequently weaponize any platforms they were sold, when possible. That’s a big caveat given the long and storied history of such weapons as the Toyota Technical, former Hilux pickups converted into DIY war machines that have been a mainstay in asymmetric conflicts since the ’80s.    

“We also pledge to explore the development of technological features that could mitigate or reduce these risks,” the group continued, but “to be clear, we are not taking issue with existing technologies that nations and their government agencies use to defend themselves and uphold their laws.” They also call on policymakers as well as the rest of the robotics development community to take up similar pledges. 

White House unveils its ‘blueprint’ for an AI Bill of Rights

Between Amazon using tech to extract more productivity from its workforce, Clearview AI harvesting our facial features, schools trying to scan children’s rooms before exams and Facebook’s whole “accused of contributing to genocide” thing, the same technologies that have brought us the wonders of the modern world have also brought about some of the horrors of the modern world. And, apparently, the Biden Administration isn’t going to stand for it.

U.S. President Joe Biden gestures at a robot dog called 'Spot' at the Robotics Lab, North Carolina Agricultural and Technical State University, Harold L. Martin Engineering Research and Innovation Complex in Greensboro, North Carolina, U.S., April 14, 2022. REUTERS/Leah Millis
Leah Millis / reuters

On Tuesday, the White House Office of Science and Technology Policy (OSTP) released its long-awaited Blueprint for an AI Bill of Rights (BoR). The document will, “help guide the design, development, and deployment of artificial intelligence (AI) and other automated systems so that they protect the rights of the American public,” per a White House press release.

As such, the BoR will advocate for five principles: Safe and Effective Systems, Algorithmic Discrimination Protections, Data Privacy, Notice and Explanation, and Human Alternatives, Consideration, and Fallback. “Simply put, systems should work, they shouldn’t discriminate, they shouldn’t use data indiscriminately,” BoR co-writer Suresh Venkatasubramanian, wrote in a tweet thread Tuesday. “They should be visible and easy to understand, and they shouldn’t eliminate human interlocutors.”

“There were thousands of edits and comments that made the document strong, rich, and detailed,” Venkatasubramanian continued. “The AI Bill of Rights reflects, as befits the title, a consensus, broad, and deep American vision of how to govern the automated technologies that impact our lives.” 

“Automated technologies are driving remarkable innovations and shaping important decisions that impact people’s rights, opportunities, and access. The Blueprint for an AI Bill of Rights is for everyone who interacts daily with these powerful technologies — and every person whose life has been altered by unaccountable algorithms,” said Office of Science and Technology Policy Deputy Director for Science and Society Dr. Alondra Nelson. “The practices laid out in the Blueprint for an AI Bill of Rights aren’t just aspirational; they are achievable and urgently necessary to build technologies and a society that works for all of us.”

The Administration has spent more than a year developing the BoR to its current state, including extensive public outreach through panel discussions, public listening sessions, and meetings with everyone from workers and activists to CEOs and entrepreneurs. In addition to the bill itself, the OSTP has also released a companion work, From Principles to Practice, which details concrete steps for both government and NGO entities, public and private companies alike, take to ensure they are operating within the scope and spirit of the document. 

“Effectively implementing these processes require the cooperation of and collaboration among industry, civil society, researchers, policymakers, technologists, and the public,” the BoR reads. Of course, the blueprint details no actual enforcement mechanisms.

Hitting the Books: What the wearables of tomorrow might look like

Apple’s Watch Ultra, with its 2000-nit digital display and GPS capabilities, is a far cry from its Revolutionary War-era self-winding forebears. What sorts of wondrous body-mounted technologies might we see another hundred years hence? In his new book, The Skeptic’s Guide to the Future, Dr. Steven Novella (with assists from his brothers, Bob and Jay Novella) examines the history of wearables and the technologies that enable them to extrapolate where further advances in flexible circuitry, wireless connectivity and thermoelectric power generation might lead.

Skeptic's Guide to the Future Cover
Grand Central Publishing

Excerpted from the book The Skeptics’ Guide to the Future: What Yesterday’s Science and Science Fiction Tell Us About the World of Tomorrow by Dr. Steven Novella, with Bob Novella and Jay Novella. Copyright © 2022 by SGU Productions, Inc. Reprinted with permission of Grand Central Publishing. All rights reserved. 


Technology that Enables Wearables

As the name implies, wearable technology is simply technology designed to be worn, so it will advance as technology in general advances. For example, as timekeeping technology progressed, so did the wristwatch, leading to the smartwatches of today. There are certain advances that lend themselves particularly to wearable technology. One such development is miniaturization.

The ability to make technology smaller is a general trend that benefits wearables by extending the number of technologies that are small enough to be conveniently and comfortably worn. We are all familiar by now with the incredible miniaturization in the electronics industry, and especially in computer chip technology. Postage-stamp-sized chips are now more powerful than computers that would have filled entire rooms in prior decades.

As is evidenced by the high-quality cameras on a typical smartphone, optical technology has already significantly miniaturized. There is ongoing research into tinier optics still, using metamaterials to produce telephoto and zoom lenses without the need for bulky glass.

“Nanotechnology” is now a collective buzzword for machines that are built at the microscopic scale (although technically it is much smaller still), and of course, nanotech will have incredible implications for wearables.

We are also at the dawn of flexible electronics, also called “flex circuits” and more collectively “flex tech.” This involves printing circuits onto a flexible plastic substrate, allowing for softer technology that moves as we move. Flexible technology can more easily be incorporated into clothing, even woven into its fabric. The advent of two-dimensional materials, like carbon nanotubes, which can form the basis of electronics and circuits, are also highly flexible. Organic circuits are yet another technology that allows for the circuits to be made of flexible material, rather than just printed on flexible material.

Circuits can also be directly printed onto the skin, as a tattoo, using conductive inks that can act as sensors. One company, Tech Tats, already offers one such tattoo for medical monitoring purposes. The ink is printed in the upper layers of the skin, so they are not permanent. They can monitor things like heart rate and communicate this information wirelessly to a smartphone.

Wearable electronics have to be powered. Small watch batteries already exist, but they have finite energy. Luckily there are a host of technologies being developed that can harvest small amounts of energy from the environment to power wearables (in addition to implantable devices and other small electronics). Perhaps the earliest example of this was the self-winding watch, the first evidence of which comes from 1776. Swiss watchmaker Abraham-Louis Perrelet developed a pocket watch with a pendulum that would wind the watch from the movement of normal walking. Reportedly it took about fifteen minutes of walking to be fully wound.

There are also ways to generate electric power that are not just mechanical power. Four types of ambient energy exist in the environment—mechanical, thermal, radiant (e.g., sunlight), and chemical. Piezoelectric technology, for example, converts applied mechanical strain into electrical current. The mechanical force can come from the impact of your foot hitting the ground, or just from moving your limbs or even breathing. Quartz and bone are piezoelectric materials, but it can also be manufactured as barium titanate and lead zirconate titanate. Electrostatic and electromagnetic devices harvest mechanical energy in the form of vibrations.

There are thermoelectric generators that can produce electricity from differences in temperature. As humans are warm-blooded mammals, a significant amount of electricity can be created from the waste heat we constantly shed. There are also thermoelectric generators that are made from flexible material, combining flex tech with energy harvesting. This technology is mostly in the prototype phase right now. For example, in 2021, engineers published the development of a flexible thermoelectric generator made from an aerogel-silicone composite with embedded liquid metal conductors resulting in a flexible that could be worn on the wrist and could generate enough electricity to power a small device.

Ambient radiant energy in the form of sunlight can be converted to electricity through the photoelectric effect. This is the basis of solar panels, but small and flexible solar panels can be incorporated into wearable devices as well.

All of these energy-harvesting technologies can also double as sensing technology—they can sense heat, light, vibration, or mechanical strain and produce a signal in response. Tiny self-powered sensors can therefore be ubiquitous in our technology.

The Future of Wearable Tech

The technology already exists, or is on the cusp, to have small, flexible, self-powered, and durable electronic devices and sensors, incorporated with wireless technology and advanced miniaturized digital technology. We therefore can convert existing tools and devices into wearable versions, or use them to explore new options for wearable tech. We also can increasingly incorporate digital technology into our clothing, jewelry, and wearable equipment. This means that wearable tech will likely increasingly shift from passive objects to active technology integrated into the rest of our digital lives.

There are some obvious applications here, even though it is difficult to predict what people will find useful versus annoying or simply useless. Smartphones have already become smartwatches, or they can pair together for extended functionality. Google Glass is an early attempt at incorporating computer technology into wearable glasses, and we know how it has been received.

If we extrapolate this technology, one manifestation is that the clothing and gear we already wear can be converted into electronic devices we already use, or they can be enhanced with new functionality that replaces or supports existing devices.

We may, for example, continue to use a smartphone as the hub of our portable electronics. Perhaps that smartphone will be connected not only to wireless earbuds as they are now, but also to a wireless monitor built into glasses, or sensors that monitor health vitals or daily activity. Potentially, the phone could communicate with any device on the planet, so it could automatically contact your doctor’s office regarding any concerning changes, or contact emergency services if appropriate.

Portable cameras could also monitor and record the environment, not just for documenting purposes but also to direct people to desired locations or services, or contact the police if a crime or disaster is in progress.

As our appliances increasingly become part of the “internet of things,” we too will become part of that internet through what we wear, or what’s printed on or implanted beneath our skin. We might, in a very real sense, become part of our home, office, workplace, or car, as one integrated technological whole.

We’ve mostly been considering day-to-day life, but there will also be wearable tech for special occupations and situations. An extreme version of this is exosuits for industrial or military applications. Think Iron Man, although that level of tech is currently fantasy. There is no portable power source that can match Iron Man’s arc reactor, and there doesn’t appear to be any place to store the massive amounts of propellant necessary to fly as he does.

More realistic versions of industrial exosuits are already a reality and will only get better. A better sci-fi analogy might be the loader exo-suit worn by Ripley in Aliens. Powered metal exosuits for construction workers have been in development for decades. The earliest example is the Hardiman, developed by General Electric between 1965 and 1971. That project essentially failed and the Hardiman was never used, but since then development has continued. Applications have mostly been medical, such as helping people with paralysis walk. Industrial uses are still minimal and do not yet include whole-body suits. However, such suits can theoretically greatly enhance the strength of workers, allowing them to carry heavy loads. They could also incorporate tools they would normally use, such as rivet guns and welders.

Military applications for powered exosuits would likely include armor, visual aids such as infrared or night-vision goggles, weapons and targeting systems, and communications. Such exosuits could turn a single soldier into not just enhanced infantry, but also a tank, artillery, communications, medic, and mule for supplies.

Military development might also push technology for built-in emergency medical protocols. A suit could automatically apply pressure to a wound to reduce bleeding. There are already pressure pants that prevent shock by helping to maintain blood pressure. More ambitious tech could automatically inject drugs to counteract chemical warfare, increase blood pressure, reduce pain, or prevent infection. These could be controlled by either onboard AI or remotely by a battlefield medic who is monitoring the soldiers under their watch and taking actions remotely through their suits.

Once this kind of technology matures, it can then trickle down to civilian applications. Someone with life-threatening allergies could carry epinephrine on them to be injected, or they could wear an autoinjector that will dose them as necessary, or be remotely triggered by an emergency medical responder.

Everything discussed so far is an extrapolation from existing technology, and these more mature applications are feasible within fifty years or so. What about the far future? This is likely where nanotechnology comes in. Imagine wearing a nanosuit that fits like a second skin but that is made from programmable and reconfigurable material. It can form any mundane physical object you might need, on command. Essentially, the suit would be every tool ever made.

You could also change your fashion on demand. Go from casual in the morning to business casual for a meeting and then formal for a dinner party without ever changing your clothes. Beyond mere fashion, this could be programmable cosplay—do you want to be a pirate, or a werewolf? More practically, such a nanoskin could be well ventilated when it’s warm and then puff out for good insulation when it’s cold. In fact, it could automatically adjust your skin temperature for maximal comfort.

Such material can be soft and comfortable, but bunch up and become hard when it encounters force, essentially functioning as highly effective armor. If you are injured, it could stem bleeding, maintain pressure, even do chest compressions if necessary. In fact, once such a second skin becomes widely adopted, life without it may quickly become unimaginable and scary.

Wearable technology may become the ultimate in small or portable technology because of the convenience and effectiveness of being able to carry it around with us. As shown, many of the technologies we are discussing might converge on wearable technology, which is a good reminder that when we try to imagine the future, we cannot simply extrapolate one technology but must consider how all technology will interact. We may be making our wearables out of 2D materials, powered by AI and robotic technology, with a brain-machine interface that we use for virtual reality. We may also be creating customized wearables with additive manufacturing, using our home 3D printer.

Tesla debuts an actual, mechanical prototype of its Optimus robot

It seems like just yesterday that Elon Musk ushered a person in a spandex suit onto the Tesla AI Day 2021 stage and told us it was a robot — or at least would probably be one eventually. In the intervening 13 months, the company has apparently been hard at work, replacing the squishy bits from what crowd saw on stage with proper electronics and mechanizations. At this year’s AI Day on Friday, Tesla unveiled the next iteration of its Optimus robotics platform and, well, at least there isn’t still a person on the inside? 

tesla bot
Tesla

Tesla CEO Elon Musk debuted the “first” Optimus (again, skinny guy in a leotard, not an actual machine) in August of last year and, true to his nature, proceeded to set out a series of increasingly incredible claims about the platform’s future capabilities — just like how the Cybertruck will have unbreakable windows. As Musk explained at the time, the Optimus will operate an AI similar to the company’s Autopilot system (the one that keeps chasing stationary ambulances) and be capable of working safely around humans without extensive prior training. 

Additionally, the Tesla Bot would understand complex verbal commands, Musk assured the assembled crowd, it would have “human-level hands,” be able to both move at 5 MPH and carry up to 45 pounds despite standing under six feet tall and weighing 125 pounds. And, most incredibly, Tesla would have a working prototype for all of that by 2022, which brings us to today.

production  tesla bot
Tesla

Kicking off the event, CEO Elon Musk was joined almost immediately on stage by an early development platform prototype of the robot — the very first time one of the test units had walked unassisted by an umbilical tether. Lacking any exterior panelling to reveal the Tesla-designed actuators inside, the robot moved at a halting and ponderous pace, not unlike early Asimos and certainly a far cry from the deft acrobatics that Boston Robotics’ Atlas exhibits.

Tesla Bot
Tesla

The Tesla team also rolled out a further developed, but still tethered iteration as well, pictured above. “it wasn’t quite ready to walk,” Musk said, “but I think we’ll walk in a few weeks. We wanted to show you the robot that’s actually really close to what is going to production.” 

Tesla Bot
Tesla

“Our goal is to make a useful humanoid robot as quickly as possible,” Musk said. “And we’ve also designed it using the same discipline that we use in designing the car, which is to say… to make the robot at an high volume at low cost with higher reliability.” He estimates that they could cost under $20,000 when built at volume. 

The Optimus will be equipped with a 2.3 kWh battery pack which integrates the various power control systems into a single PCB. That should be sufficient to get the robot through a full day of work, per Tesla’s engineering team which joined Musk on stage during the event. 

Tesla Bot
Tesla

“Humans are also pretty efficient at somethings but not so efficient at other times,” Lizzie Miskovetz, a Senior Mechanical Design Engineer at Tesla, and a member of the engineering team explained. While humans can sustain themselves on small amounts of food, we cannot halt our metabolisms when not working. 

“On the robot platform, what we’re going to do is we’re going to minimize that. Idle power consumption, drop it as low as possible,” she continued. The team also plans to strip as much complexity and mass as possible from the robot’s arms and legs. “We’re going to reduce our part count and our power consumption of every element possible. We’re going to do things like reduce the sensing and the wiring at our extremities,” Miskovetz said. 

Tesla Bot
Tesla

What’s more, expensive and heavy materials will be swapped out with plastics that trade slight losses in stiffness with larger savings in weight. “We are carrying over most of our designing experience from the car to the robot,” Milan Kovac, Tesla’s Director of Autopilot Software Engineering said. 

To enable the Optimus to move about in real world situations, “We want to leverage both the autopilot hardware and the software for the humanoid platform, but because it’s different in requirements and inform factor,” Miskovetz said. “It’s going to do everything that a human brain does: processing vision data, making split-second decisions based on multiple sensory inputs and also communications,” thanks to integrated Wi-Fi and cellular radios.

“The human hand has the ability to move at 300 degrees per second, as tens of thousands of tactile sensors. It has the ability to grasp and manipulate almost every object in our daily lives,” Kovac said. “We were inspired by biology. [Optimus hands] have five fingers and opposable thumb. Our fingers are driven by metallic tendons that are both flexible and strong because the ability to complete wide aperture power grasps while also being optimized for precision, gripping of small, thin and delicate objects.” 

Tesla Bot
Tesla

Each hand will offer 11 degrees of freedom derived from its six dedicated actuators, as well as “complex mechanisms that allow the hand to adapt to the objects being grasped.” Kovac said. “We [also] have a non-backdrivable finger drive. This clutching mechanism allows us to hold and transport objects without having to turn on the hand motors.”

“We’re starting out having something that’s usable,” Kovac concluded, “but it’s far from being useful. It’s still a long and exciting road ahead of us.” Tesla engineering plans to get the enclosed, production iteration up and walking around without a tether in the next few weeks, then begin exploring more real-world applications and tangible use cases the Optimus might wind up in. 

“After seeing what we’ve shown tonight,” Kovac said. “I’m pretty sure we can get this done within the next few months or years and maybe make this product a reality and change the entire economy.”

AI is already better at lip reading than we are

They Shall Not Grow Old, a 2018 documentary about the lives and aspirations of British and New Zealand soldiers living through World War I from acclaimed Lord of the Rings director Peter Jackson, had its hundred-plus-year-old silent footage modernized through both colorization and the recording of new audio for previously non-existent dialog. To get an idea of what the folks featured in the archival footage were saying, Jackson hired a team of forensic lip readers to guesstimate their recorded utterances. Reportedly, “the lip readers were so precise they were even able to determine the dialect and accent of the people speaking.”

“These blokes did not live in a black and white, silent world, and this film is not about the war; it’s about the soldier’s experience fighting the war,” Jackson told the Daily Sentinel in 2018. “I wanted the audience to see, as close as possible, what the soldiers saw, and how they saw it, and heard it.”

That is quite the linguistic feat given that a 2009 study found that most people can only read lips with around 20 percent accuracy and the CDC’s Hearing Loss in Children Parent’s Guide estimates that, “a good speech reader might be able to see only 4 to 5 words in a 12-word sentence.” Similarly, a 2011 study out of the University of Oklahoma saw only around 10 percent accuracy in its test subjects.

“Any individual who achieved a CUNY lip-reading score of 30 percent correct is considered an outlier, giving them a T-score of nearly 80 three times the standard deviation from the mean. A lip-reading recognition accuracy score of 45 percent correct places an individual 5 standard deviations above the mean,” the 2011 study concluded. “These results quantify the inherent difficulty in visual-only sentence recognition.”

For humans, lip reading is a lot like batting in the Major Leagues — consistently get it right even just three times out of ten and you’ll be among the best to ever play the game. For modern machine learning systems, lip reading is more like playing Go — just round after round of beating up on the meatsacks that created and enslaved you — with today’s state-of-the-art systems achieving well over 95 percent sentence-level word accuracy. And as they continue to improve, we could soon see a day where tasks from silent-movie processing and silent dictation in public to biometric identification are handled by AI systems.

Context matters

it's a statue
Wikipedia / Public Domain

Now, one would think that humans would be better at lip reading by now given that we’ve been officially practicing the technique since the days of Spanish Benedictine monk, Pedro Ponce de León, who is credited with pioneering the idea in the early 16th century.

“We usually think of speech as what we hear, but the audible part of speech is only part of it,” Dr. Fabian Campbell-West, CTO of lip reading app developer, Liopa, told Engadget via email. “As we perceive it, a person’s speech can be divided into visual and auditory units. The visual units, called visemes, are seen as lip movements. The audible units, called phonemes, are heard as sound waves.”

“When we’re communicating with each other face-to-face is often preferred because we are sensitive to both visual and auditory information,” he continued. “However, there are approximately three times as many phonemes as visemes. In other words, lip movements alone do not contain as much information as the audible part of speech.”

“Most lipreading actuations, besides the lips and sometimes tongue and teeth, are latent and difficult to disambiguate without context,” then-Oxford University researcher and LipNet developer, Yannis Assael, noted in 2016, citing Fisher’s earlier studies. These homophemes are the secret to Bad Lip Reading’s success.

What’s wild is that Bad Lip Reading will generally work in any spoken language, whether it’s pitch-accent like English or tonal like Vietnamese. “Language does make a difference, especially those with unique sounds that aren’t common in other languages,” Campbell-West said. “Each language has syntax and pronunciation rules that will affect how it is interpreted. Broadly speaking, the methods for understanding are the same.”

“Tonal languages are interesting because they use the same word with different tone (like musical pitch) changes to convey meaning,” he continued. “Intuitively this would present a challenge for lip reading, however research shows that it’s still possible to interpret speech this way. Part of the reason is that changing tone requires physiological changes that can manifest visually. Lip reading is also done over time, so the context of previous visemes, words and phrases can help with understanding.”

“It matters in terms of how good your knowledge of the language is because you’re basically limiting the set of ambiguities that you can search for,” Adrian KC Lee, ScD, Professor and Chair of the Speech and Hearing Sciences Department, Speech and Hearing Sciences at University of Washington, told Engadget. “Say, ‘cold; and ‘hold,’ right? If you just sit in front of a mirror, you can’t really tell the difference. So from a physical point of view, it’s impossible, but if I’m holding something versus talking about the weather, you, by the context, already know.”

In addition to the general context of the larger conversion, much of what people convey when they speak comes across non-verbally. “Communication is usually easier when you can see the person as well as hear them,” Campbell-West said, “but the recent proliferation of video calls has shown us all that it’s not just about seeing the person there’s a lot more nuance. There is a lot more potential for building intelligent automated systems for understanding human communication than what is currently possible.”

Missing a forest for the trees, linguistically

While human and machine lip readers have the same general end goal, the aims of their individual processes differ greatly. As a team of researchers from Iran University of Science and Technology argued in 2021, “Over the past years, several methods have been proposed for a person to lip-read, but there is an important difference between these methods and the lip-reading methods suggested in AI. The purpose of the proposed methods for lip-reading by the machine is to convert visual information into words… However, the main purpose of lip-reading by humans is to understand the meaning of speech and not to understand every single word of speech.”

In short, “humans are generally lazy and rely on context because we have a lot of prior knowledge,” Lee explained. And it’s that dissonance in process — the linguistic equivalent of missing a forest for the trees — that presents such a unique challenge to the goal of automating lip reading.

“A major obstacle in the study of lipreading is the lack of a standard and practical database,” said Hao. “The size and quality of the database determine the training effect of this model, and a perfect database will also promote the discovery and solution of more and more complex and difficult problems in lipreading tasks.” Other obstacles can include environmental factors like poor lighting and shifting backgrounds which can confound machine vision systems, as can variances due the speaker’s skin tone, the rotational angle of their head (which shifts the viewed angle of the mouth) and the obscuring presence of wrinkles and beards.

As Assael notes, “Machine lipreading is difficult because it requires extracting spatiotemporal features from the video (since both position and motion are important).” However, as Mingfeng Hao of Xinjiang University explains in 2020’s A Survey on Lip Reading Technology, “action recognition, which belongs to video classification, can be classified through a single image.” So, “while lipreading often needs to extract the features related to the speech content from a single image and analyze the time relationship between the whole sequence of images to infer the content.“ It’s an obstacle that requires both natural language processing and machine vision capabilities to overcome.

Acronym soup

Today, speech recognition comes in three flavors, depending on the input source. What we’re talking about today falls under Visual Speech Recognition (VSR) research — that is, using only visual means to understand what is being conveyed. Conversely, there’s Automated Speech Recognition (ASR) which relies entirely on audio, ie “Hey Siri,” and Audio-Visual Automated Speech Recognition (AV-ASR), which incorporates both audio and visual cues into its guesses.

“Research into automatic speech recognition (ASR) is extremely mature and the current state-of the-art is unrecognizable compared to what was possible when the research started,” Campbell-West said. “Visual speech recognition (VSR) is still at the relatively early stages of exploitation and systems will continue to mature.” Liopa’s SRAVI app, which enables hospital patients to communicate regardless of whether they can actively verbalize, relies on the latter methodology. “This can use both modes of information to help overcome the deficiencies of the other,” he said. “In future there will absolutely be systems that use additional cues to support understanding.”

“There are several differences between VSR implementations,” Campbell-West continued. “From a technical perspective the architecture of how the models are built is different … Deep-learning problems can be approached from two different angles. The first is looking for the best possible architecture, the second is using a large amount of data to cover as much variation as possible. Both approaches are important and can be combined.”

In the early days of VSR research, datasets like AVLetters had to be hand-labeled and -categorized, a labor-intensive limitation that severely restricted the amount of data available for training machine learning models. As such, initial research focused first on the absolute basics — alphabet and number-level identification — before eventually advancing to word- and phrase-level identification, with sentence-level being today’s state-of-the-art which seeks to understand human speech in more natural settings and situations.

In recent years, the rise of more advanced deep learning techniques, which train models on essentially the internet at large, along with the massive expansion of social and visual media posted online, have enabled researchers to generate far larger datasets, like the Oxford-BBC Lip Reading Sentences 2 (LRS2), which is based on thousands of spoken lines from various BBC programs. LRS3-TED gleaned 150,000 sentences from various TED programs while the LSVSR (Large-Scale Visual Speech Recognition) database, among the largest currently in existence offers 140,000 hours of audio segments with 2,934,899 speech statements and over 127,000 words.

And it’s not just English: Similar datasets exist for a number of languages such as HIT-AVDB-II, which is based on a set of Chinese poems, or IV2, a French database composed of 300 people saying the same 15 phrases. Similar sets exist too for Russian, Spanish and Czech-language applications.

Looking ahead

VSR’s future could wind up looking a lot like ASR’s past, says Campbell-West, “There are many barriers for adoption of VSR, as there were for ASR during its development over the last few decades.” Privacy is a big one, of course. Though the younger generations are less inhibited with documenting their lives on line, Campbell-West said, “people are rightly more aware of privacy now then they were before. People may tolerate a microphone while not tolerating a camera.”

Regardless, Campbell-West remains excited about VSR’s potential future applications, such as high-fidelity automated captioning. “I envisage a real-time subtitling system so you can get live subtitles in your glasses when speaking to someone,” Campbell-West said. “For anyone hard-of-hearing this could be a life-changing application, but even for general use in noisy environments this could be useful.”

“There are circumstances where noise makes ASR very difficult but voice control is advantageous, such as in a car,” he continued. “VSR could help these systems become better and safer for the driver and passengers.”

On the other hand, Lee, whose lab at UW has researched Brain-Computer Interface technologies extensively, sees wearable text displays more as a “stopgap” measure until BCI tech further matures. “We don’t necessarily want to sell BCI to that point where, ‘Okay, we’re gonna do brain-to-brain communication without even talking out loud,’“ Lee said. “In a decade or so, you’ll find biological signals being leveraged in hearing aids, for sure. As little as [the device] seeing where your eyes glance may be able to give it a clue on where to focus listening.”

“I hesitate to really say ‘oh yeah, we’re gonna get brain-controlled hearing aids,” Lee conceded. “I think it is doable, but you know, it will take time.”