Tag Archives: technology management

Photo by David Travis on Unsplash

Habit Forming Platforms Part I

In my previous blog I had talked about Technology Evolution & touched upon how we have seen waves come in and go. I am converting that into a series of posts. First of the many posts related to Habit forming products & platforms. It captures my reflections around customer engagement mostly inspired from my readings of the book by Nir Eyal : HOOKED : How To Build Habit-Forming Products 

Nir Eyal is an Israeli-born American author, lecturer and investor known for his bestselling book, Hooked: How to Build Habit-Forming Products.  He teaches and has expertise in areas of psychology, technology & business.

Everybody you meet , there is always a common thread on talks of how to improve customer engagement. I also do realize that we are trying to make sincere efforts to improve it all the time but still keep failing at it ! It is important to retrospect why this is the case and why do we keep losing engagement from our customers , not making the value proposition compelling enough to keep their attention live and fresh !

When I started reading the book , it became very clear to me how forming habits is imperative for the survival of many products. The current pandemic is a living example where the consumption patterns and habits are rapidly shaping to create survivability  , continuity , & pivoting away from the pandemic.

Back in 2001 , when I joined the industry internet was coming out of womb and world was still about rich desktop applications. Some of us would remember Power Builder front-ends on Windows ! People at that time would expect the technology on web to be just like that , comparison point of totally then divergent tool sets ! there was expectation that web should replicate every aspect of experience there by underscoring the other tranformational impact of internet.  It was a struggle on how to manage this transition with scores of teams involved trying to get this right .  the books offers a set of learning on how such situations should be addressed from a platforms stand-point.

  1. Companies need to change behavior by presenting users with an implicit choice between old and new.
  2. Platform services should be enjoyable for the sake of its customers.
  3. Building Platforms that are marginally better than others will never shake the old habits of customers , with broad adoption base.

A classic paper by John Gourville , a professor of marketing at Harvard Business School stipulates that

Many innovations fail because consumers irrationally overvalue old while companies irrationally overvalue the new.

As we build platforms

  • We need to be better by  miles to even stand a chance for customers to get hooked to us.
  • If the platform and products require high degree of behavior change , then they are doomed to fail even if the benefits of using the new product are clear and substantial !
  • We need dramatic improvement to our software design or restatement of problem to break the users out of their old routines. 

Quoting another example from the book is that of the QWERTY keyboard , which was developed in 1870s ! Simply putting this layout prevented users from jamming metal type bars of early machines. Many people have tried to since then reinvent keyboards and relate it to better ergonomics BUT QWERTY still remains a standard. How does it survive ?

For a simple reason that there is very high costs attached to changing the user behavior and challenge the stored value for it within its customers. The whole process of relearning and adopting stands little or very less chance of success!

Business heads , platform architects , designers & developers need to:

  • Engage
  • Gauge
  • Modify

to make important decisions regarding how platform should be developed to trigger engagement for customers to get hooked to it.

We will talk in upcoming blog posts more around how to challenge and change the stored value in customer’s mind in order to increase likelihood of adoption. In the mean time , if you have any feedback or comments , please do share !

 

Scaling Analytics With Agility Part II

This is last in the two part series where I have have tried to explain approaches to achieving agility with data. If you have not already gone through part I , then follow this link.

Reminder of what we are trying to achieve by adopting any one or hybrid approach is as follows: 

  • Optimize Query performance 
  • Common Query Language 
  • Central data model for business analysts
  • Fast access to data insights 

The part – I of this series helped us understand the single Physical Data Store approach and now we are going to talk about Logical Data Store Approach

Logical Data Store Approach

In this approach we do not execute a Load of data to single store but tend to hand off more directly to data analysts ability to construct logical view or data models across various data sources without the need of lifting and shifting the data. There is a need to construct logical data models and to a large extent removes the need of developers to get involved straight up in any process.

capture

The above landscape tells us that Single Data Store architecture does provide some inhibitions to agility at the end of the day and this is something which logical data ware house architecture is looking to address.

Typical Architecture

The main theme here is that we are centralizing the data models as opposed to the data itself.

Let us now summarize the approaches across both major themes to achieve agility:

Considerations to Single Physical Data Store Approach

Pros

  • Brings data to one place and then use the store to do transformations

  • Takes an approach where the lake contains all relevant information in raw state post ingestion on continuous basis to cater to multiple personas

  • If used in conjunction to ELT architecture, it provides for a fine balance between developer and analyst community. The schematization of raw data is helpful and allows analysts to create logical data models post transformation within the store

  • Extent of development required depends on choice of ELT infrastructure adopted

  • It is not a hard choice or decision of CTO’s organization and in essence with less engineering resources you may still achieve quite a lot

Cons

  • It is dependent on the architecture that the teams would have followed in bringing data to a single store, implying that if customer connector architecture or ETL approach has been adopted with wrong choices then, the friction to get data in the store will remain very high
  • Storage of data and connecting to DWH will determine pricing of bring it all together along with other investments to standardize the ingestion pipeline architecture

Considerations to Logical Data Store Approach

Pros

  • It centralizes the data modelling and not the actual raw data store
  • It centralizes the modeled data for BI exposure
  • It provides for more self-service BI architecture

Cons

  • Maturity of organization and type of skill set to operate this kind of infrastructure
  • At what size should this be recommended?
  • How much help would be required for multiple businesses become self-serve on this model?
  • The CTO organization can make a choice for this but would need Data Ops to work alongside BI for creating & enabling data models that allow you to operate and leverage the power or else this can get reduced to being just another ELT infra that may not justify its deployment

Summary

Through this mini-series , one would get general idea of various methods by which agility can be achieved to unlock the golden joins ( as I call it ) that drives maximum value for the organization and provides data when it is needed most. 

According to me in order to make a choice , try to introspect and define the maturity index of following three parameters

  • Analyst Org
  • Engineering Org
  • Current DWH infrastructure
  • Budget 
  • Data set sizes 

In addition to this also be reminded that hybrid approach will always bean option if the organization is quite large and centralization in general to drive all the personas might not fit through one or the other working model.

 

Scaling Analytics With Agility Part I

Design principles and guiding forces for achieving unified analytics in the world of distributed data sources can vary. I thought it might be a good idea to just digitize some of my thoughts & where does it make sense to bring them all together and what are the trade-offs in doing so. In a multi-part series we will explore some approaches and then analyse what parameters are necessary to measure in order to pick an approach.

The common goal which we are always trying to search is geared towards any one of all of the following in combination:

  • Optimize Query performance 
  • Common Query Language 
  • Central data model for business analysts
  • Fast access to data insights 

Analyst Use Cases

While you can have many ways of looking at analytics , I generally tend classify things in two buckets to keep it simple. 

Report or Dashboard View

Capture

Non-Dashboard view

  • Combine data to generate insights, or to do data scientific activities which drive marketing behavior
  • Process for pulling data together in the lake to analyze and create marketing pipelines.
  • Online / offline share, share on complaints and sample orders
  • Combining touchpoints => is customer searching on site, then contacting customer care and placing an order via e-mail => what kind of product?

Single Physical Data Store Approach

This approach requires you house all your data in place and follows the paradigm of building analytics on top of single warehouse technology

Data Ingestion Approach

Data ingestion approach is driven by adopting custom connectors or ELT architecture that allows you to get the data to your central data store

Custom Connectors

This is a very traditional approach where developers internally work using any programming language to run batch mode extractors and bring a highly developer centric approach to developing / deploying extractors. It lacks standards of extraction architecture and does not follow a templated based connector architecture paradigm. This approach comes with least flexibility and agility into ingesting data with agility into your store. The custom connectors basically serve as ELT pipelines and are prone to continuous upkeep.

Standard ELT Pipelines

Industry offers many standard ELT pipelines and these platforms are standardized as architecture approach to provide for wide variety of connectors. Two most popular ELT architecture platforms are Stitch and FiveTran

There would be more and other ways , I am not contesting that but trying to just convey the pipeline flow and certain things can be achieved.

Stitch

It has been around for quite some time in market and now has been acquired by Talend , it is a blend of following traits

  • Provides for standard connectors certified by Stitch, these are around 90+
  • Provides for standard Tap-Target architecture which is open source. Read more about it at singer.io
  • Offers Schematization in standard as well open architecture development
  • Has limited exposure of Google related connectors and meta information
  • You can control historical import of information
  • Fosters open source development
  • Great community support
  • Has got a good User Experience
  • It is now backed by a world leader in data pipeline architecture
Skillset requirements

As a Data Analyst you can deal with this Stitch easily, while if you do not have a connector then you can develop one using Pyhton skillset using a standards-based approach as offered by Singer and get certified by Stitch. Using Stitch Import API in conjunction to Lambda functions also allows you to send data to Stitch.

Stitch Approaches Summarized
Using Stitch’s Standard Connectors

Stitch supports more than 90 integrations

Using Stitch’s Import API

If building and maintaining a script to retrieve and push data from Google Search Console is feasible for you or your team, Stitch’s Import API integration can be used as a receiving point for JSON or TRANSIT posts that would then be loaded to the destination warehouse connected to your Stitch account.

Singer Development

Stitch also works with integrations built as part of our open source project Singer. Singer includes a specification that allows integrations to be built by anyone, and our team provides support to members of the community who want to contribute code to the project via the Singer Slack group.

Any community built integrations can also be submitted by their creator to our team for review and potential inclusion in Stitch as a Community Integration using this form. Otherwise, integrations can be run within your infrastructure with the Stitch target and directed toward an Import API integration within your account.

If you or a member of your team is interested in building a Singer integration, for Google Search Console or otherwise, I would recommend checking out our getting started guide, and bringing any development-focused questions to the Singer Slack group.

Sponsored Development

If this is especially time-sensitive or building an in-house solution isn’t feasible for your team, Stitch’s Enterprise plan can include arrangements for the development of custom integrations that are built to ensure your requirements are met.

Typical Architecture

Capture

FiveTran

This is a new age ELT pipeline platform that focused on bring rich schematization & large connector architecture base to its users. It is blend of following traits

  • Provides very large connector base that covers almost all tools available
  • Is continuously evolving
  • Offers rich Schematization
  • Boasts of handling very large dataset with optimality
  • Is highly optimized to Snowflake
  • Comes with multiple destinations architecture
  • Provides for event stream ingestion architecture
  • API driven economy is available but evolving
  • Has in-depth exposure of Google related data stores
Skillset requirements

As a Data Analyst you can deal Fivetran easily. It is touted more as a data analyst friendly platform and while developers can get involved using cloud functions architecture, this is not something that is considered as an open source standard, you need to define and architect it as per the needs of the FiveTran platform.

Using FiveTran’s Standard Connectors

Leverage 100+ connectors from FiveTran

Using FiveTran’s Cloud Function Architecture

This is achieved using cloud function architecture where you need to deploy cloud functions on their platform and make that connector available for consumption

Sponsored Development

This is possible using enterprise contract

Typical Architecture

capture-2.png

Summary

I explained data centralization approaches using above facts and in next part I will continue to talk about virtual datawarehouse architecture and what kind of benefits it might entail.

Vertical Decomposition Of Software Systems

This is a very light post on vertical decomposition and by no means is most comprehensive read on this topic! It is meant for engineers and product managers that are looking after or working in such initiative to resurrect their current approach or pick something new from the examples.

Vertical decomposition powered by micro-services is a handy approach while trying to break down applications which appear to be black-boxed and contain objects with too many responsibilities. This complex overlap causes unwanted extension in data structures and creates convoluted schema that albeit powers the application but fails to provide right level of elasticity for absorbing ever changing needs of business.

Vertical decomposition can help you break down the application that encapsulates single business domain such as order , search , customer , product etc. In turn these single business domains should have concrete resource definitions with right level of granularity powered by micro-services to expose their

  • Functions
  • Configuration
  • Events
  • Discoverbility

While looking at vertical decomposition , one needs to spend time and decide which pieces need to be decomposed and how an incremental migration of same would take place.

Some great examples where you will find a particular resource having multiple responsibilities is as follows [ not comprehensive but enough to drive home the point ] :

  • Cross Domain
    • Customer carries definition on discounting that is linked to products
    • Customer specific products are linked to the product model
  • Complex Structure
    • Customer carries definition of tiers with logic and rules attached to it
    • Ordering pattern combines buying and promotions together

Such scenarios clearly help you identify radius of operation and allow you to start breaking down aspects of complexities with primarily two things as drivers:

  • Break things which are simple in nature and help you lay foundation
    • Example : [ assigning loyalty tiers to customers ]
    • Break Loyalty program out from the customer
  • Identify dependencies that reduce reliance on your current monolith [ follow anti-corruption design in case dependency exists ]
    • Example : [ login service ]
  • Carve out independent services that can expose required functionality without going back to monolith
    • Example : [ Separate discount from customer as product discount or separate promotions from buying operations ]

If you are some where above and started to experiment then for sure your delivery teams are comfortable with building micro-services and ready to go. However you may hit roadblock as the deconstruction process is not an easy one. Domains are over-lapped and need to break down that one resource which does it all. One of the key things to achieve that is start imagining distributed systems and finding a “best-of”breed” approach to separate by keeping persons in mind mapped to domains and then to best of breed architecture. This way you would be able to spread the system as  a platform and created separation of concerns which can then start to shape up the action map for the persona and domain.

Achieving this is not a one day’s job and requires us to ramp up on many fronts as an organization. The safety net in doing so should be always kept in mind so that you can ensure to isolate or promote basis the phase outcomes before we proceed. The whole exercise will start with broad funnel approach and then start to narrow down eventually leading to spurt of services that control separate domains and talk to each other in a contractual approach.

Tale Of Many Internal applications

Internal tooling architecture is as much an integral part of tech strategy as the core tech roadmap which enables topline. Often I have seen that we make tooling without understanding the customers ( internal ). These tools largely tend to grow as isolated patches and considered low on agenda to maintain or service…

Internal tools and utilities should also have coding standards with backbone which allows them to keep pace with the main architectural platform..within organization various business stacks should be made to plugin into that so that we can systematize software inventory and support the business operations in a scalable manner. Best of breed approach with services based coupling should help organization create business efficient operations.

It may not look that savvy to build and maintain internal applications but it has indirect ramifications to core business and people of the organization. Internal tooals usually get created with dev. mindset and generally ignore the customer who would use it because adoption is through imposition and approach is to rid development from iterative non-engineering tasks.

Tooling is great project for freshers but it should have boundaries of organization architecture on top of which they are able to create plugin based applications for business processes or core functions that are enabled by mainline architecture.