What's the Big Deal About Big Data?

Raw personal data is now being used to fine-tune health, make decisions and improve romantic relationships.

Big Data -- that is, the amassing and analysis of gigantic amounts of raw information from a wide range of sources -- might sound scary to people who imagine government supercomputers sifting through phone call records and toll-booth surveillance video footage to track their daily movements, or malevolent corporate giants siphoning up details of their web browsing and credit card purchases to subtly manipulate them into buying certain products.

But not all of our civilization's vast reservoir of personal data is being gathered by the National Security Agency or Amazon.com.

Increasing numbers of people are gathering amassing vast amounts of information about themselves. Case in point: Economist data editor Ken Cukier, co-author of the 2013 book "Big Data: A Revolution That Will Transform How We Live, Work, and Think," who wears a small device on his wrist that measures health-related data, ranging from how many steps he takes each day to how much sleep he gets.

"If you wanted to measure your sleep patterns 15 years ago, you had to go to a university lab and have them hook up $100,000 worth of sensors and equipment," he explained. "Today, you can do it with a band on your wrist for $100."

Eventually, Cukier hopes that his data, aggregated with other device wearers, will provide a precise answer to the question of how exercise affects sleep. "If you exercise more, do you sleep better, or worse? Or is it that you sleep less when you exercise, but it's more profound sleep? Before, we didn't really have enough information to pose the right questions. But now we will."

Cukier belongs to a seemingly growing group of DIY collectors of Big Data about themselves, sometimes called self-trackers or members of something called the quantified self (QA) movement.

The Pew Internet and American Life Project reports that 7 percent of U.S. adults use a smart phone app to continuously gather and tabulate some sort of health-related data about themselves, such as diet, blood pressure, sleep patterns, or frequency of headaches. But within that group there's a smaller, undefined slice who use phones, computers, wearable sensors and sophisticated software apps to amass minute data on everything from their daily movements, exercise routines and social contacts to fluctuations in their emotional states and how many cups of iced coffee they drink.

By analyzing the data and turning it into visual charts and graphs, some are striving to fine-tune their health, boost their work productivity, and even improve their romantic relationships. Others simply are indulging their self-curiosity, and hoping to understand themselves a little better.

Harvard University social psychologist Justin J. Lehmiller thinks that the personal Big Data movement is an outgrowth of a natural human tendency to engage in self-perception. "We look to our own behaviors to determine who we are," he said. "We can infer how we feel about something, based upon how we behave toward it."

Already, organizations and companies are springing up to cater to personal Big Data users. One example is Quantified Self, founded in 2008 by Wired magazine journalists Kevin Kelly and Gary Wolf. QS is holding an annual conference in San Francisco this week in which 400 self-trackers, makers of tools for Big Data, and researchers will gather to hear presentations with titles such as "The Journey to Existence Mapping," "Are You What You Eat or What You Buy?" and "What I Learned from 30 Days of Continuous ECG." The last is a talk by a self-tracker who wore a cardiac monitor continuously for a month, so that she could gather a data set about her body's physiological patterns.

In a TED talk in 2012, QS co-founder Wolf explained that amassing and analyzing personal data is useful "when we reflect, learn, remember and want to improve ... if we want to act more effectively in the world, we have to get to know ourselves better."

The self-tracking trend actually has historical roots. Perhaps one of the first gatherers of Big Data was Benjamin Franklin, who carried around a small journal, containing spreadsheets upon which he marked down the days when he transgressed against virtues such as chastity, sincerity and humility, as well as when he ate or drank to excess.

Today, Franklin might carry a smart phone and conduct more detailed and accurate self-surveillance and upload the data to a website, noting whatever behaviors he wanted to track and then tweeting notes on them to your.flowing.data.com, a self-tracking and analysis website created by UCLA statistics graduate student Nathan Yau.

"We make tiny choices every day," Yau wrote in the site's FAQ. "Those choices become habits, and those habits develop into behaviors. your.flowingdata helps you record these choices."

Another personal Big Data proponent, graphic designer Nicholas Felton, co-founded the online self-tracking and visualization tool Daytum.com, whose goal is to provide users with tools "to examine and communicate their habits and routines."

Felton, who now works for Facebook, has compiled and published the equivalent of corporate annual reports about his life activities. His 2010-2011 biannual report, for example, reveals that he worked 2,395.5 hours during that period, walked 1,309.5 miles, spent the equivalent of 8.5 days riding the New York City subway, traveled to 40 cities in six countries, consumed 251 cups of his favorite beverage (ice coffee), listened to 26,015 songs, and spent more time with family members in March and July than he did in January or August.

Another self-quantifier, actress Lisa Betts-LaCroix, told Newsweek that she gathered data on events ranging from marital arguments to times that she cooked dinner for her husband, in an effort to strengthen their relationship.

In addition to smart phones and computers, Big Data users often employ even more sophisticated gadgets to gather intimate information about themselves, such as 3D accelerometers that will track a person's movements, and the Fitbit, a sort of Swiss Army Knife for self-trackers that measures caloric expenditure, weight and even sleep patterns.

Social psychologist Lehmiller said that self-tracking, in addition to helping people manage their health better, could benefit them by providing more clarity and confidence in their feelings. On the downside, "it could make life less fun. If you're constantly obsessing and worrying about the behaviors that you're monitoring, you're not going to have as much fun in life. And it could be stressful, as well."

Daytum co-founder Fenton wrote in a September 2013 essay for FT.com that "I'd like to believe that collecting data about myself has changed my lifestyle for the better, but it doesn't always work that way," and he sometimes takes Big Data to extremes. Recently, for example, he started collecting data on the number of teeth lost by his cat, "which I accept probably does sound a little weird to some people."

But Cukier notes that personal Self Data is still at the 1.0 stage.

Within a few years, he expects that personal monitoring devices will become so cheap and unobtrusive -- "imagine a sensor the size of a grain of salt, embedded in your earlobe, that will gather data on five vital signs" -- that they'll be ubiquitous.

"What today looks like a bunch of fitness freaks and narcissists, tomorrow we'll call health care," he says. "This is how society evolves. You have an small avant garde group that eventually becomes the mainstream."

Basis Watch

Time to make up for December's bad habits by doing better in 2012. Here are the best tech tools to help you shape up and keep healthy. Who knows? You might actually keep your new year's resolution this time around. Sure, Basis can tell you time, but if you want to know your blood flow, motion, temperature, heart rate, sweat level and blood oxygen level, it'll tell you those too. With a plethora of sensors, the monitoring watch keeps an eye on your vitals, giving you an overview of health, sleep and exercise habits. Basis is an honoree for the upcoming CES Best of Innovations Design and Engineering Awards in the health and wellness category. Available for pre-order for $199. This article is part of a series about getting fit in the new year. Check out the entire Man up! feature here.

MotoActv Heart-rate Monitor

The MotoActv wants to be your personal trainer. This tiny device tells when you reach or leave your target pace, heart rate or PowerZone based on your programmed profile and goals. And to keep you going, it creates a performance playlist, pulling songs that you burned the most calories to. It also takes on a few personal assistant duties, including fetching your incoming calls and displaying on-screen text messages. Begins at $249.99.

Withings WiFi scale

For better or worse, scales don't lie. In fact, the Withings WiFi scale tells you the cold hard truth: weight, body fat percentage, and BMI. Each time you step on, it registers these stats and sends them over your home wireless network to a private Web interface. The dashboard keeps tabs on your progress with static and interactive charts. You can share this information with your doctors, personal trainers, friends and family. If you feel so inclined, you can even tweet your progress to the entire world. Available from ThinkGeek for $164.99.

BitGym Fitness Games

The average American household has 1.15 cardio machines according to the San Francisco-based health startup BitGym. But overwhelmingly, they're left to collect dust. Get ready to use the treadmill again because BitGym's iOS games are designed to keep you going. One of them, Trail Runner, shows inspiring landscapes as you're on an exercise machine, speeding up or slowing down based on your real-life workout performance. Game prices vary, but lite versions are available for free.

Runtastic App

If you prefer to run outdoors, Runtastic is an app that tracks your location, distance, time, pace and calorie consumption. It has charts that show your speed, altitude, pulse and training history. The pro version includes voice feedback, live tracking, cheering, pulse-reading, geotagging, workouts, competitions, and an integrated music player. Its iOS and Android apps have the most functionality, but Runtastic is also available on BlackBerry, Windows, and bada phones. Prices vary by device.

JayBird Freedom Earphones

The JayBird Freedom was designed for the gym rat. It uses Bluetooth connectivity, so there aren't long cords to trip over. The sound is big -- great motivation when your power track comes on. Plus, it's got enough variety of ear cushions, tips, and hooks to make sure you find the right fit; one that stays on when you're on the go.

Fitness Technologies Underwater MP3 Player

Music can motivate runners to go longer distances, why not apply the same principle to swimmers? Generally electronics and water don't mix very well, but Fitness Technologies' UWaterK7 was built for just that. The compact waterproof MP3 player debuted in the fall and will be making an appearance at CES in January. Also expected to make an appearance: the company's line of HD waterproof action cameras and waterproof stereo Bluetooth headsets. Available for $100.

Mophie Outdoor Battery Extender and Maps

Grab your iPhone. You're going for a hike. Not only does the mophie juice pack plus outdoor give you extended battery life (about 2,000 mAh, or eight hours of talk time on 3G), a corresponding app gives you access to 5 million square miles of high-resolution maps covering the continental U.S. and Hawaii. Once you download them, you no longer have to worry about losing reception. Plus the app records your progress, speed, distance, elevation, and geo-tagged photos. Available for $119.95.

Drift HD Video Camera

A good workout doesn't always mean hitting the gym. Head somewhere beautiful and find a fun activity, like biking or snowboarding. Action cams such as the Drift HD can be a good motivator to go outside. They capture amazing moments in 1080p HD video, which, upon watching, will make you want to go right back outside again. The small, light camera can be mounted to helmets or strapped on wrists and can also be controlled remotely. A night mode also means you can record in dusty or dark conditions. Feeling motivated to get your workout on? Visit our Man up! feature, chock full of info that will get your heart pumping.