As companies move more data into data stores, finding and securing specific information can be very challenging. Chandler, Arizona-based Clairvoyant is a Big Data company that has built a platform in enterprise environments called Kogni, which solves that problem.
Essentially Kogni creates a “data catalog” of sensitive data to find information that is most important for companies such as credit card numbers. Kogni uses machine learning to recognize sensitive new data as it is added. Plus Kogni can be used for issuing alerts in case sensitive information is breached.
To learn more about Clairvoyant, I interviewed CEO Chandra Ambadipudi. Ambadipudi told me that he has roots in the higher education space. Before he launched Clairvoyant, he was VP of Engineering at Apollo Education Group, which is the parent company of the University of Phoenix.
Clairvoyant works with a number of companies to build data lakes including large Hadoop players like Cloudera as well as end customers like Limelight Networks. Clairvoyant also offers data management services for customers. And Clairvoyant achieved this without external funding. Ambadipudi highlighted that because of its unique business model, Clairvoyant was able to fund product development through its services business and that helped the company grow and remain independent. In the past 5 years, revenue increased by over 60% based on a compound annual growth rate.
While working at Apollo, Ambadipudi led a team of 300 engineers, business analysts and project managers. He was also responsible for all aspects of the enterprise systems that the University of Phoenix and other Apollo subsidiaries used. He led a technical team that implemented Apollo’s first predictive analytics project code-named “P-wave,” which handles student retention. Plus he was responsible for various enterprise-class systems that included Enrollment and Admissions, student scheduling, attendance management, the SIS (Student Information System) at the center of UoP operations, Call Center operations and the first version of the mobile platform used by University of Phoenix.
Since the founding team had deep experience and expertise in software engineering and big data technologies, launching Clairvoyant seemed like a “natural choice” and a seamless move from the higher education technology space. And a clearly defined business use case helped crystallize the initial plan for the company.
“As is the case with starting and getting any new business off the ground, there were challenges. We had both a defined product idea as well a business case. A practical use case is something often missing from many tech startups. There’s a great product idea, but often no proof of concept,” said Ambadipudi in the interview. “Additionally, we were fortunate to have some senior executives and industry leaders as our advisors and mentors. This coupled with having a strong network in the Phoenix area helped us land our first customers. We also leveraged the startup community network in Chandler to get the company going. Continued focus on big data services along with product development helped us scale the company rapidly from the start.”
When Clairvoyant started in 2012, it was an engineering-focused predictive analytics company with two customers. About a year later, Clairvoyant launched a product called Blue Canary. In 2014, the company was awarded Startup of the Year by Arizona Tech Council, opened up operations in India and set up the Phoenix Data Conference. Interestingly, Blue Canary was acquired in 2015. Now Clairvoyant has over 50 customers and 300 employees around the world.
I was curious about how the acquisition of Clairvoyant’s sale of Blue Canary to Blackboard took place so I asked Ambadipudi to elaborate on that. “With our background and deep experience in the higher education vertical, we identified a business need amongst both large and small academic institutions to improve student retention. Based on the work we had done at Apollo, we realized that creating a predictive analytics driven product with custom algorithms and data science modeling tools would address this business need. This was the genesis of Blue Canary, and we created it with this specific use case. Clairvoyant as a company started with the vision to build SaaS-based products and Blue Canary was our first one. Blue Canary’s key component was a custom built Data Science Modeling tool (DataBrew) that enabled automation of predictive model building. This significantly reduced the time it took to build models from days and weeks to hours and allowed us to easily scale the product across 1000+ customers,” Ambadipudi explained.
“In November 2015, BlackBoard (owned by private equity) acquired Blue Canary. This was during a time where academic institutions were rapidly adopting technology that enabled them to improve learning and performance. BlackBoard was specifically seeking technology that could power the next generation of their BlackBoard Learning Analytics and Predict platforms. As part of the transaction, the Clairvoyant team was involved in the integration with multiple institutions across multiple data sources with different databases and APIs.”
Clairvoyant’s technology originally evolved from Java, Cassandra and Angular based products over the past five years to an engineering organization that provides solutions across multiple platforms in data security, cloud computing and Big Data. And Clairvoyant gained and “developed significant expertise in the Hadoop ecosystem building massive data ingestion pipelines, large-scale data management platforms, reliable reporting and analytic products.” Clairvoyant also manages and support large Hadoop clusters for global clients.
Upon gaining experience in helping enterprises build Big Data systems that pulled both structured and unstructured data, the company realized the risks of data breaches. And Big Data expands the compliance requirements to cover the risks associated with centralizing large volumes of data. “Many enterprises focus exclusively on perimeter security. Unfortunately, as demonstrated by an endless stream of high profile data breaches at major companies, the question of firewall breach is no longer if, but when. We developed Kogni to help solve all of these issues with tools that enable companies to discover sensitive data in enterprise data sources (cloud-based and on-premise), secure data as it is ingested, and continuously monitor data sources for possible breach and policy violations,” Ambadipudi added.
Kogni was built with three core functionalities and supporting capabilities. Kogni scans enterprise data sources, leverages machine learning and computer vision in order to identify sensitive data stored in text and images and supports a broad range of data sources like Hadoop, NoSQL, S3, and RDBMS. Kogni transparently secures data as it is ingested into Hadoop with zero code change and little performance overhead using masking, encryption, redaction and tokenization methods based on simple configuration and plugins for Sqoop, Spark, Kylo, Nifi, Streamsets, etc. And Kogni continuously monitors data sources and user behavior for anomalies and triggers alerts on sensitive data proliferation and policy violations.
Limelight Networks leverages Clairvoyant’s Hadoop Managed Services to drive improved availability, reduce outages, cut costs, and gain access to a larger pool of skilled Hadoop architects, administrators and engineers. Ambadipudi said that Limelight Networks had a complex billing and reporting use case that their existing systems were struggling to compute within target service-level agreements (SLAs). Their dataset was sensitive and for compliance reasons also needed to be isolated. And applications supporting access to the data and reporting off the data on that cluster had a user base that would be interacting with it 24 hours a day. And prior implementations took up to two weeks for these jobs to finish.
After deciding to leverage Hadoop for the implementation, Limelight Networks had to manage multiple operational challenges. For example, the stability of the Hadoop cluster was a key issue. Clairvoyant worked with Limelight to come up with an approach to design, configure, optimize and take over management of their Hadoop Infrastructure. Clairvoyant defined service-level agreements (SLAs) for availability, job completion and response times to manage the cluster with 24×7 support
Another Clairvoyant customer is Bidtellect. Bidtellect is a leader in paid content advertising distribution. As Bidtellect scaled up to 15 billion transactions per day, the company needed a cost-effective solution that could scale handle these volumes. The solution also needed to have the ability for data mining and ad-hoc query capability across a variety of additional dimensions and metrics.
Clairvoyant built a solution for Bidtellect by analyzing existing infrastructure and processes and creating a multi-tenant Hadoop based solution with lightweight data ingestion using Streamsets & Kafka. Ambadipudi said that Clairvoyant created a data model transformation from MySQL to HDFS and implemented security frameworks through encryption zones.
What are Clairvoyant’s future company goals? Ambadipudi answered that Clairvoyant’s vision is to be a “customer outcomes focused company driven by engineering excellence in the Big Data space” with plans to expand into new markets in Europe and Asia. Clairvoyant will continue focusing their services on the Hadoop ecosystem, AI and the cloud. And the product roadmap will include “Big Data vertical industry solutions built on the Kogni platform that will drive data science along with data security in specific domains.”