Tag Archives: Tech Strategy

Light-Code Data Integration With Zapier

Recently I started sharing my learning around various experiments in area of no-code & light-code. Previously I had written a blog post on no-code Airtable Integration for data collection & processing. This post is about an experiment that I did few weeks back for a Proof-Of-Concept to create tickets and search for users in Zendesk [ to many this should not need any introduction ]

In order to complete my Proof-Of-Concept , I divided my processing into four major blocks:

  • Data Entry
    • Leverages a simple app created using React & React Zapier Form
    • Deploys to a very a easy to use static web publishing platform using surge.sh
  • Data Collection & Mapping
    • Created a workflow step to collect & map data using Zapier
  • Triggers
  • Data Persistence
    • Created a workflow step to persist the processed information back into storage of choice
    • or Can also inspect the data using RequestBin

The over-all architecture flow would like somewhat like this :

flow

In terms of account set-up , you would need trial or entry level account set-up with following

  • Zapier
  • Zendesk
  • Surge.sh
  • RequestBin

In this experiment the dominant design pattern is around Zapier. As we walk through various blocks you would understand how different constructs of a Zap as Zapier calls it are at play.

Data Entry

Using a default React App , I integrated the react-zapier-form package [ details are provided above ] . This package helped me to quickly integrate with a catch-hook that was defined within the Zapier workflow which allows us to post the data from the react form to the catch-hook as a json payload.

</p>import ZapierForm from 'react-zapier-form'
 
...
 
<ZapierForm action='INSERT YOUR HOOK'>
   {({ error, loading, success }) => {
      return (
         <div>
            {!success && !loading &&
               <div>
                  <input type='email' name='Email' placeholder='Email' />
                  <textarea name='Message' placeholder='Your message' />
                  <button>Submit</button>
               </div>
            }
            {loading && <div>Loading...</div>}
            {error && <div>Something went wrong. Please try again later.</div>}
            {success && <div>Thank you for contacting us!</div>}
         </div>
      )
   }}
</ZapierForm><p class="has-text-align-justify">

Once this react app is ready for deployment , I always love to move away from localhost Proof-Of-Concept to a deployment in cloud experience , so leveraging surge.sh came very handy to that effect. Surge has been built from the ground up for native web application publishing and is committed to being the best way for Front-End Developers to put HTML5 applications into production.

& you can deploy for free for starters 🙂

</p>npm install -g surge
npm run build
cd build
mv index.html 200.html
surge<p class="has-text-align-justify">

The command sequence does as follows

  • Install surge
  • Build your React App
  • Rename index.html to 200.html [ If we don’t rename index.html, everything will work fine, but in case you have client side routing routing (maybe with React Router) and we navigate to a new route and refresh the page, we’ll get a 404 “page not found” error. Since many React projects implement client-side routing, I have included this step. If you aren’t using client-side routing, feel free to skip renaming the index.html file. Read more about adding a 200 page for client-side routing on the Surge help docs.
  • Now run the surge command , that’s it !

Data Mapping & Triggers

Zapier workflow construction is pretty straight forward and one can proceed very swiftly through the integration. As you can see that there is node based code to capture the response and then post back on a URL , which I grabbed from RequesBin to post the data.

Once the whole process runs end to end you can then see that a post of the processed information is available at the HTTP hook . One can similarly send this data to a persistent storage using Zapier as it has integrated to many popular persistence mechanism including queues.

One of the things you would see in the workflow schematic image and the workflow itself is the use of a request_id that is generated on client side and then floated across the processing pipeline for us to create trace all along Zapier workflow and then be able to get the result look-up using the same request_id. I used the uuid package to achieve this piece of GUID generation.

I hope people find this useful for their day-to-day problem statement around workflow automation and it provides them some more options on how to move steadily through some integration problems of connecting with different Apps because Zapier provides more than 1500+ integrations that can be useful to automate many tasks.

If you have any feedback or comments post back on the blog . Happy Reading !

No-Code Airtable Integration

I have been using Airtable for quite sometime now at RecipeDabba where I work as part-time co-founder and coder ! My wife Rakshita Dwivedi , is the actual consumer of my work.

Almost every feature that is described by Airtable helps to power light weight tech-support that for my wife’s 21-day challenges in multiple formats that helps promote her healthy eating philopshy for kids. This became ever more significant during pandemic as she shifted bulk of her work online.

The diagram which you see below has been architected is powered using Airtable to create a workflow based architecture:

Schematic flow – copyright – Recipedabba

Airtable is a versatile cloud based sheet / database solution that helps automate large part of light weight process through

  • Multiple data types
  • Formulae
  • Blocks
  • Forms

I use all of the above in combination to do multiple pieces in the workflow like

  • Basics
    • Table Creations
    • Views
  • Data Grouping
    • Use of filters , group by
  • Analytics & Derivations
    • Roll-up fields [ very power full feature ]
    • Formulae to derive new fields [ this was another awesome feature ]
  • Data Entry
    • Forms
  • Blocks
    • De-Dupe Checks
    • Charts
Chart Presentation of Data

You can see above how the table data is quickly transformed into a basic chart visulization.

De-dupe block to remove duplicate enteries

An awesome block to remove duplicate entries from the system , with few clicks and configurations.

Snippets from the form view

Form rendition on mobile and desktop is very nie . Since we started to use this , the mothers [ who are primary collector of informaiton on behalf of kids who particpate ] , have found it easy to fill information and send it back to us!

Formulae and Applications

We can work on top of the data and apply many conditioanlities , thus allowing a flexible viewing of data in real time. Some of these things can take coding effort while connecting with analytics but , first level aggregation and analytics on daily basis has been very easy to perform in Airtable.

Overall for a upcoming or very small set-up Airtable . If you want to know more about how to do things in Airtable , feel free to ping me via comments and I will see if I can help !

Scaling Analytics With Agility Part II

This is last in the two part series where I have have tried to explain approaches to achieving agility with data. If you have not already gone through part I , then follow this link.

Reminder of what we are trying to achieve by adopting any one or hybrid approach is as follows: 

  • Optimize Query performance 
  • Common Query Language 
  • Central data model for business analysts
  • Fast access to data insights 

The part – I of this series helped us understand the single Physical Data Store approach and now we are going to talk about Logical Data Store Approach

Logical Data Store Approach

In this approach we do not execute a Load of data to single store but tend to hand off more directly to data analysts ability to construct logical view or data models across various data sources without the need of lifting and shifting the data. There is a need to construct logical data models and to a large extent removes the need of developers to get involved straight up in any process.

capture

The above landscape tells us that Single Data Store architecture does provide some inhibitions to agility at the end of the day and this is something which logical data ware house architecture is looking to address.

Typical Architecture

The main theme here is that we are centralizing the data models as opposed to the data itself.

Let us now summarize the approaches across both major themes to achieve agility:

Considerations to Single Physical Data Store Approach

Pros

  • Brings data to one place and then use the store to do transformations

  • Takes an approach where the lake contains all relevant information in raw state post ingestion on continuous basis to cater to multiple personas

  • If used in conjunction to ELT architecture, it provides for a fine balance between developer and analyst community. The schematization of raw data is helpful and allows analysts to create logical data models post transformation within the store

  • Extent of development required depends on choice of ELT infrastructure adopted

  • It is not a hard choice or decision of CTO’s organization and in essence with less engineering resources you may still achieve quite a lot

Cons

  • It is dependent on the architecture that the teams would have followed in bringing data to a single store, implying that if customer connector architecture or ETL approach has been adopted with wrong choices then, the friction to get data in the store will remain very high
  • Storage of data and connecting to DWH will determine pricing of bring it all together along with other investments to standardize the ingestion pipeline architecture

Considerations to Logical Data Store Approach

Pros

  • It centralizes the data modelling and not the actual raw data store
  • It centralizes the modeled data for BI exposure
  • It provides for more self-service BI architecture

Cons

  • Maturity of organization and type of skill set to operate this kind of infrastructure
  • At what size should this be recommended?
  • How much help would be required for multiple businesses become self-serve on this model?
  • The CTO organization can make a choice for this but would need Data Ops to work alongside BI for creating & enabling data models that allow you to operate and leverage the power or else this can get reduced to being just another ELT infra that may not justify its deployment

Summary

Through this mini-series , one would get general idea of various methods by which agility can be achieved to unlock the golden joins ( as I call it ) that drives maximum value for the organization and provides data when it is needed most. 

According to me in order to make a choice , try to introspect and define the maturity index of following three parameters

  • Analyst Org
  • Engineering Org
  • Current DWH infrastructure
  • Budget 
  • Data set sizes 

In addition to this also be reminded that hybrid approach will always bean option if the organization is quite large and centralization in general to drive all the personas might not fit through one or the other working model.

 

Scaling Analytics With Agility Part I

Design principles and guiding forces for achieving unified analytics in the world of distributed data sources can vary. I thought it might be a good idea to just digitize some of my thoughts & where does it make sense to bring them all together and what are the trade-offs in doing so. In a multi-part series we will explore some approaches and then analyse what parameters are necessary to measure in order to pick an approach.

The common goal which we are always trying to search is geared towards any one of all of the following in combination:

  • Optimize Query performance 
  • Common Query Language 
  • Central data model for business analysts
  • Fast access to data insights 

Analyst Use Cases

While you can have many ways of looking at analytics , I generally tend classify things in two buckets to keep it simple. 

Report or Dashboard View

Capture

Non-Dashboard view

  • Combine data to generate insights, or to do data scientific activities which drive marketing behavior
  • Process for pulling data together in the lake to analyze and create marketing pipelines.
  • Online / offline share, share on complaints and sample orders
  • Combining touchpoints => is customer searching on site, then contacting customer care and placing an order via e-mail => what kind of product?

Single Physical Data Store Approach

This approach requires you house all your data in place and follows the paradigm of building analytics on top of single warehouse technology

Data Ingestion Approach

Data ingestion approach is driven by adopting custom connectors or ELT architecture that allows you to get the data to your central data store

Custom Connectors

This is a very traditional approach where developers internally work using any programming language to run batch mode extractors and bring a highly developer centric approach to developing / deploying extractors. It lacks standards of extraction architecture and does not follow a templated based connector architecture paradigm. This approach comes with least flexibility and agility into ingesting data with agility into your store. The custom connectors basically serve as ELT pipelines and are prone to continuous upkeep.

Standard ELT Pipelines

Industry offers many standard ELT pipelines and these platforms are standardized as architecture approach to provide for wide variety of connectors. Two most popular ELT architecture platforms are Stitch and FiveTran

There would be more and other ways , I am not contesting that but trying to just convey the pipeline flow and certain things can be achieved.

Stitch

It has been around for quite some time in market and now has been acquired by Talend , it is a blend of following traits

  • Provides for standard connectors certified by Stitch, these are around 90+
  • Provides for standard Tap-Target architecture which is open source. Read more about it at singer.io
  • Offers Schematization in standard as well open architecture development
  • Has limited exposure of Google related connectors and meta information
  • You can control historical import of information
  • Fosters open source development
  • Great community support
  • Has got a good User Experience
  • It is now backed by a world leader in data pipeline architecture
Skillset requirements

As a Data Analyst you can deal with this Stitch easily, while if you do not have a connector then you can develop one using Pyhton skillset using a standards-based approach as offered by Singer and get certified by Stitch. Using Stitch Import API in conjunction to Lambda functions also allows you to send data to Stitch.

Stitch Approaches Summarized
Using Stitch’s Standard Connectors

Stitch supports more than 90 integrations

Using Stitch’s Import API

If building and maintaining a script to retrieve and push data from Google Search Console is feasible for you or your team, Stitch’s Import API integration can be used as a receiving point for JSON or TRANSIT posts that would then be loaded to the destination warehouse connected to your Stitch account.

Singer Development

Stitch also works with integrations built as part of our open source project Singer. Singer includes a specification that allows integrations to be built by anyone, and our team provides support to members of the community who want to contribute code to the project via the Singer Slack group.

Any community built integrations can also be submitted by their creator to our team for review and potential inclusion in Stitch as a Community Integration using this form. Otherwise, integrations can be run within your infrastructure with the Stitch target and directed toward an Import API integration within your account.

If you or a member of your team is interested in building a Singer integration, for Google Search Console or otherwise, I would recommend checking out our getting started guide, and bringing any development-focused questions to the Singer Slack group.

Sponsored Development

If this is especially time-sensitive or building an in-house solution isn’t feasible for your team, Stitch’s Enterprise plan can include arrangements for the development of custom integrations that are built to ensure your requirements are met.

Typical Architecture

Capture

FiveTran

This is a new age ELT pipeline platform that focused on bring rich schematization & large connector architecture base to its users. It is blend of following traits

  • Provides very large connector base that covers almost all tools available
  • Is continuously evolving
  • Offers rich Schematization
  • Boasts of handling very large dataset with optimality
  • Is highly optimized to Snowflake
  • Comes with multiple destinations architecture
  • Provides for event stream ingestion architecture
  • API driven economy is available but evolving
  • Has in-depth exposure of Google related data stores
Skillset requirements

As a Data Analyst you can deal Fivetran easily. It is touted more as a data analyst friendly platform and while developers can get involved using cloud functions architecture, this is not something that is considered as an open source standard, you need to define and architect it as per the needs of the FiveTran platform.

Using FiveTran’s Standard Connectors

Leverage 100+ connectors from FiveTran

Using FiveTran’s Cloud Function Architecture

This is achieved using cloud function architecture where you need to deploy cloud functions on their platform and make that connector available for consumption

Sponsored Development

This is possible using enterprise contract

Typical Architecture

capture-2.png

Summary

I explained data centralization approaches using above facts and in next part I will continue to talk about virtual datawarehouse architecture and what kind of benefits it might entail.

Be The Thinker In System

In the modern world knowing only one thing does not suffice and it holds true across the board. The area in product management seems to become the biggest testimony to this aspect. If you want to be a strategic contributor as a product manager then in that case it becomes important that one should first become the thinker in system. I recommend everybody taking this path to definitely add to your reading list the following book : Think In Systems  by Donella H. Meadows

Coming back to the main point , what I did like to reflect here is that today, you can’t host your product on one platform. Usually, your product lives on multiple platforms or even more the platform itself is an net-sum product suite —your website, partner API , connecting systems etc.  Product managers need the experience and know-how to manage one product and features across all platforms. There’s whole art and science of keeping your product aligned and synced across diverse platforms.

One needs to be build understanding by observing and mapping the System, for example using the well-known method of User Journey Mapping.  They’re valuable as a way to describe your understanding of a system, and are best applied in collaboration with your critical stakeholders. The value of mapping and modelling the systems you’re trying to influence is that it helps you to spot the trends and patterns within that system.

Focus on following aspects:

Flow

What information is shown to who, how is that information shown, and who can manipulate information.

Rules

What governs the systems and how it operates. The bounded and unbounded context. The macro effects on the system.

Mindset

Changing behavior is not a joke but encouraging a new approach or adoption is an important point that needs to be kept in mind of a product manager to be a strategic thinker.

How do you organize the development of the product to match the way users consume it split up across different experiences? 

µ-services – Enabling Agile Development

I got an opportunity to share my insights in CIOReview Mag. – March 17 edition.

You are either building software… or losing to someone who is – @littleidea

Technology plays a pivotal role today on how we deliver to our business objectives. The choice of technology can determine over a period of time how business will scale and disrupt the market. One of the most modern paradigms of support new technology evolution has been Microservices.

I have tried to provide simple insight through this articles what it means to operate in API economy and opportunities it spells along with challenges.

mag

Happy reading!

Monolith 2 Microservices @ OSI2016

I have been little inactive for past few weeks & somewhat busy doing too many things !

I spent two days in Bengaluru participating as well speaking at Open Source India 2016. It was  a great experience and lovely audience to interact with. I had chosen to speak on microservices and how this is an evolving paradigm in current business context.

Idea of my talk was to drive following point:

You are either building software… or losing to someone who is – @littleidea

Comparison to monolith and the journey to microservices is just a paradigm shift depending on what we are trying to achieve and whether or not we have spine to carry that shift. It is not necessary to make a change if you can follow good programming practices and still remain monolithic in nature. Understanding why one would want to move to microservices and how is very important . Realising & mapping what intrinsic and extrinsic value addition it will do a over a period of time is the necessary business bridge to achieve synergy with current business objectives.

At the end of the day if one has to be great at building software then following things must always be kept in mind:

  • Technology
    • Build technology for ease of business
  • People
    • Hire smart and talented people across various domains
  • Product
    • Provide products as business enablers
  • Data
    • Create visibility of insights for remaining competitive
  •  Go-To-Market
    • Disrupt Go-To-Market strategies

One needs to build technology by hiring domain agnostic talented people who can build products to enable business through data and power disruptive go-to-markets.

You can find my presentation in following location: microservices @ OSI @2016

Happy Reading!

Deferred Objects & Arrays

Just documented some stuff around deferred objects and array…

Was coding just for fun and encountered the issue of not getting back the objects provided to the deferred’s resolve() method as jQuery calls the done() and fail()callbacks with individual parameters, not an array. That means we have to use the arguments pseudo-array to get all the resolved/rejected objects returned by the array of deferreds.

$.when.apply($,deferreds).then(function() {
var objects=arguments; // The array of resolved objects as a pseudo-array
...
};

Here is a solution inspired by when.js‘s when.all() method that addresses these problems:

// Put somewhere in your scripting environment
if (jQuery.when.all===undefined) {
    jQuery.when.all = function(deferreds) {
        var deferred = new jQuery.Deferred();
        $.when.apply(jQuery, deferreds).then(
            function() {
                deferred.resolve(Array.prototype.slice.call(arguments));
            },
            function() {
                deferred.fail(Array.prototype.slice.call(arguments));
            });

        return deferred;
    }
}

Now you can simply pass in an array of deferreds/promises and get back an array of resolved/rejected objects in your callback, like so:

$.when.all(deferreds).then(function(objects) {
    console.log("Resolved objects:", objects);
});

Refer : Pass in an array of Deferreds to $.when()