{"id":25717,"date":"2021-04-13T17:34:43","date_gmt":"2021-04-13T16:34:43","guid":{"rendered":"https:\/\/www.intercom.com\/blog\/?p=25717"},"modified":"2021-05-12T14:24:24","modified_gmt":"2021-05-12T13:24:24","slug":"building-elasticsearch-at-intercom","status":"publish","type":"post","link":"https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/","title":{"rendered":"Search at Intercom: Building our Elasticsearch cloud on AWS"},"content":{"rendered":"<p>In Intercom, we believe that shipping is our company\u2019s heartbeat. Our product engineers are empowered to build great features, fast.<\/p>\n<p>A large part of making this belief a reality is the idea of <a href=\"https:\/\/www.intercom.com\/blog\/run-less-software\/\">running less software.<\/a>\u00a0 Every product team is burdened by the effort required to evolve and maintain complex infrastructure that powers their features. For this reason, we chose to run exclusively on AWS and wherever possible, we make use of battle-tested AWS services, be it RDS Aurora for our relational databases, the Simple Queue Service (SQS) for our async workers or ElastiCache for our caching layer.<\/p>\n<p>However, as with any rule, there comes an exception, a case that cannot fit neatly into the principles we&#8217;ve outlined as a company. So how do we handle these exceptions? In Intercom, we were forced to answer these questions as we built out our search functionalities and embraced <a href=\"https:\/\/www.intercom.com\/blog\/upgrading-elasticsearch\/\">Elasticsearch<\/a>.<\/p>\n<p><em>This post is a version of a talk I gave at <a href=\"https:\/\/community.elastic.co\/events\/details\/elastic-global-community-presents-community-conference-2021\/\">Elastic Community Conference<\/a> recently. You can watch a video of that talk below, or read on to learn how we built our Elasticsearch cloud on AWS.<\/em><\/p>\n<div class=\"oembed-wrapper oembed--youtube\"><iframe loading=\"lazy\" title=\"ElasticCC: Building your Elasticsearch cloud on AWS using EC2, SSM and Lambda\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/qBWM9eEuPsQ?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/div>\n<h2 id=\"core-part-of-intercom\">Core part of Intercom<\/h2>\n<p>Search is an indispensable part of Intercom. It underpins a whole host of core Intercom features \u2013 <a href=\"https:\/\/www.intercom.com\/blog\/help\/en\/articles\/4323882-see-a-custom-list-of-conversations-in-real-time-with-inbox-views\">Inbox Views<\/a>, <a href=\"https:\/\/developers.intercom.com\/intercom-api-reference\/reference\">API<\/a>, <a href=\"https:\/\/www.intercom.com\/blog\/articles\">Articles<\/a>, the user list, <a href=\"https:\/\/www.intercom.com\/blog\/announcing-conversation-topics-custom-reports\/\">Reporting<\/a>, <a href=\"https:\/\/www.intercom.com\/blog\/campaign\/resolution-bot\">Resolution Bot<\/a>, and our internal logging systems. Our search clusters contain more than 200TB of customer data, store more than 100 billion documents and, on at the peak on an average day, serve more than 13 million queries every minute. It does not just power the free text search functionality either \u2013 our Reporting pipeline is based on rich data aggregations running on Elasticsearch and our User, Message and Conversation matching features allow customers to perform complex predicate-based lookups against hundreds of data attributes and millions of documents using our search layer.<\/p>\n<blockquote class=\"pullquote-style-one\"><p>&#8220;Our search clusters contain more than 200TB of customer data, store more than 100 billion documents and can serve more than 13 million queries every minute&#8221;<\/p><\/blockquote>\n<p>At the time when we were building many of these features, there simply didn\u2019t exist external Elasticsearch services that could give us the necessary dials to control this layer at scale, quality, and performance that we needed. So we decided to do something contrary to our engineering principles \u2013 run our search infrastructure ourselves on Elastic Compute Cloud hosts, or EC2 as it&#8217;s known. Doing this meant we also took on the responsibility of ensuring the product teams who use it and the infrastructure teams who maintain it have a great experience with a minimum of effort. This might be an exception to our &#8220;run less software&#8221; philosophy, but we were determined that it should never feel like it. We wanted it to feel seamless and invisible.<\/p>\n<h2 id=\"three-areas-of-focus\">Three areas of focus<\/h2>\n<p>When we started the work of transparently automating away the biggest sources of frustration caused by managing search infrastructure, we decided to focus on three most expensive areas: applying security patches, performing software upgrades, and replacing failed EC2 instances.<\/p>\n<p>All three, and security patching especially, occur frequently and so are very time expensive to perform for a product team. Stateless, ephemeral infrastructure like a set of web services running on EC2 can be rebuilt or upgraded relatively quickly by recycling the hosts, but stateful infrastructure like Elasticsearch is a different beast.<\/p>\n<blockquote class=\"pullquote-style-one\"><p>&#8220;How do you take a data- and infrastructure-heavy service like Elasticsearch and build automation around it that makes all that heavy maintenance as invisible as possible?&#8221;<\/p><\/blockquote>\n<p>Taking down data-heavy instances takes a long time as data might need to be relocated for clusters to remain available. This can cause surges in network traffic, destabilize the cluster and open the door to random failures in the relatively long window during which the operation is being performed.<\/p>\n<p>So, how do you take a data- and infrastructure-heavy service like Elasticsearch and build automation around it that makes all that heavy maintenance as invisible as possible?<\/p>\n<p>To start with, you need a mechanism to safely take a node out of the cluster without compromising the availability of data. Depending on the need, Elasticsearch provides two suitable mechanisms that can be used for this purpose:<\/p>\n<p>If a node needs to be taken out of the cluster for brief maintenance (say replacing the version of the <a href=\"https:\/\/linuxhint.com\/introduction_linux_kernel\/\">Linux kernel<\/a> with a new one), Elasticsearch can use the rolling restart mechanism and mark the data as temporarily \u201cunassigned.\u201d No data movement takes place if the maintenance can be performed in a short enough time. This is achieved by setting the cluster.routing.allocation.enable setting to false for the duration of the work:<\/p>\n<p><span style=\"background-color: #fdf1db;\"><code> PUT _cluster\/settings<br \/>\n{<br \/>\n\"persistent\": {<br \/>\n\"cluster.routing.allocation.enable\": \u201cnone\u201d<br \/>\n}<br \/>\n}<br \/>\n<\/code><\/span><\/p>\n<p>If a node needs to be completely removed from the cluster (due to hardware issues, for example), we can tell Elasticsearch to safely exclude the node and reallocate the data elsewhere while keeping the cluster fully available. Copied data is marked as \u201cauthoritative\u201d only when the operation finishes:<\/p>\n<p><span style=\"background-color: #fdf1db;\"><code>PUT _cluster\/settings<br \/>\n{<br \/>\n\"transient\" : {<br \/>\n\"cluster.routing.allocation.exclude._ip\" : \"&lt;node-ip&gt;\"<br \/>\n}<br \/>\n}<br \/>\n<\/code><\/span><\/p>\n<p>Now, the second part of the story is, once you know about these mechanisms, how can you write automation that safely scales them up to a dozen or so clusters running hundreds of nodes between them?<\/p>\n<p>First, you need to provide some prerequisites and formal state checks for every operation to make your automation safe:<\/p>\n<ul>\n<li>Make sure the cluster is healthy and serving traffic both before and after each node is processed. Operations should not proceed further if the cluster is destabilized at any one time.<\/li>\n<li>Make sure that nodes are transparently removed from any load balancers and customer traffic safely drained away to other nodes to avoid customer impact and safely put back into rotation once the operation is complete.<\/li>\n<\/ul>\n<h2 id=\"providing-invisible-management\">Providing invisible management<\/h2>\n<p>To provide the needed level of this \u201cinvisible\u201d management, we utilized a combination of two technologies \u2013 <a href=\"https:\/\/aws.amazon.com\/opsworks\/chefautomate\/\">Chef<\/a> and <a href=\"https:\/\/aws.amazon.com\/systems-manager\/\">AWS Systems Manager<\/a> (SSM). Chef is a widely used configuration management framework with plenty of community support and was a good engineering fit as it allows you to write lifecycle automation in a dialect of Ruby, one of the core languages Intercom is based on. We decided to use it to power the node-level automated lifecycle of EC2 instances as they join and leave the cluster, including things like Java and Elasticsearch installation, automatic provisioning and tuning of ephemeral SSDs that host the Elasticsearch data and more.<\/p>\n<p>While Chef was good for managing the state of individual nodes, we needed a better tool for managing the co-ordinated cluster-wide workflows described above. SSM is an infrastructure control service that AWS themselves use for managing their vast EC2 resources via custom-made scripts called \u201ccommands\u201d, and so came with good guarantees on scale and robustness. It provided some important advantages on top of Chef:<\/p>\n<ul>\n<li>Cron-like automated maintenance windows can be specified to run our workflow, so we can have automation running during off-peak hours.<\/li>\n<li>Flexible instance discovery using tags allows targeting only specific groups of instances. For example, we split the ES primary and replica data (called \u201c<a href=\"https:\/\/www.intercom.com\/blog\/database-architecture-sharding-project\/\">shards<\/a>\u201d) per availability zone (AZ) to minimize the blast radius and let the system perform updates per AZ, always leaving the cluster in a healthy state.<\/li>\n<li>We implemented complex automation as Ruby tools and easily tied them into SSM commands, but also more importantly into GitHub and our CI\/CD system allowing us to treat automation at the same level of operational excellence as we do our production codebase. This allows the actual code of the SSM command to be a thin wrapper that pulls the repository master, deploys, and executes the necessary command.<\/li>\n<\/ul>\n<p>SSM executes the commands using an on-instance daemon agent called AWS SSM Agent. This architecture has both an advantage (it makes it very robust) and a downside in that there is no \u201coutside\u201d place that is executing your automation and thus you cannot target nodes that are experiencing failures or are turned off. For this purpose we extended the system to enqueue such operations (for example, deleting a failed or a stopped node from the cluster) to a simple external AWS Lambda function driven via a dedicated SQS queue.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2021\/04\/ElasticSearch-Architecture01.jpg\" \/><\/p>\n<p style=\"text-align: center;\"><em>Architecture of the system<\/em><\/p>\n<p>The system is integrated with Datadog, which our product teams use for monitoring, allowing tracking key metrics via easy-to-use dashboards, and Slack, which we use for intra-company communication, to proactively inform teams on actions being executed, ensuring high levels of operational visibility.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2021\/04\/ElasticSearch-Automation-Workflows.jpg\" \/><\/p>\n<p style=\"text-align: center;\"><em>Key automation workflows<\/em><\/p>\n<h2 id=\"saving-time-money-and-effort\">Saving time, money, and effort<\/h2>\n<p>This system allowed us to achieve massive time savings in performing infrastructure maintenance. For instance, when the <a href=\"https:\/\/www.theverge.com\/2019\/5\/14\/18623708\/zombieload-attack-intel-processors-speculative-execution\">ZombieLoad vulnerability<\/a> hit the IT world in 2019, we just shrugged off the cost of the rollout \u2013 automation simply applied the fix across hundreds of Elasticsearch nodes as part of kernel security updates with no engineer involvement necessary.<\/p>\n<blockquote class=\"pullquote-style-two\"><p>&#8220;By thoughtfully accepting the cost of this exception and by conscientiously iterating on it, we were able to bring our Elasticsearch service back in line with our principles&#8221;<\/p><\/blockquote>\n<p>Keeping up with <a href=\"https:\/\/www.intercom.com\/blog\/upgrading-elasticsearch\/\">Elasticsearch versions<\/a> to make sure Intercom can tap into the latest and greatest search features would previously require costly maintenance operations. Not any more \u2013 at the push of a button, we can upgrade the cluster version with zero downtime. We currently run all of our clusters on ES 7.9, just a few minor versions shy of the latest.<\/p>\n<p>Finally, the ability to do a live change of node EC2 types provided significant cost savings and ensured we can easily right-size any of our search clusters to achieve the best price\/performance ratio.<\/p>\n<p>It wasn\u2019t easy, but by thoughtfully accepting the cost of this exception and by conscientiously iterating on it, we were able to bring our Elasticsearch service back in line with our principles.\u00a0 This has allowed us greater flexibility without sacrificing the kind of product we like to build.<\/p>\n\n","protected":false},"excerpt":{"rendered":"<p>At Intercom, we rely on Elasticsearch as a core technology to power key parts of our platform. But how do we make it a seamless experience for our product teams? Here&#8217;s how we run our search infrastructure on AWS.<\/p>\n","protected":false},"author":475,"featured_media":25716,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"category":[12898],"tags":[14178,146],"coauthors":[24857],"class_list":["post-25717","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engineering","tag-elasticsearch","tag-recruitment"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.3 (Yoast SEO v27.3) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Search at Intercom: Building our Elasticsearch cloud on AWS - The Intercom Blog<\/title>\n<meta name=\"description\" content=\"Elasticsearch is a core technology at Intercom, and here&#039;s how we run our search infrastructure on AWS using EC2 hosts.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Search at Intercom: Building our Elasticsearch cloud on AWS\" \/>\n<meta property=\"og:description\" content=\"Elasticsearch is a core technology at Intercom, and here&#039;s how we run our search infrastructure on AWS using EC2 hosts.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/\" \/>\n<meta property=\"og:site_name\" content=\"The Intercom Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/intercominc\" \/>\n<meta property=\"article:published_time\" content=\"2021-04-13T16:34:43+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-12T13:24:24+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2021\/04\/Hero-Elasticsearch-1800.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1800\" \/>\n\t<meta property=\"og:image:height\" content=\"842\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Andrej Blagojevi\u0107\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@intercom\" \/>\n<meta name=\"twitter:site\" content=\"@intercom\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Andrej Blagojevi\u0107\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/building-elasticsearch-at-intercom\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/building-elasticsearch-at-intercom\\\/\"},\"author\":{\"name\":\"Andrej Blagojevi\u0107\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#\\\/schema\\\/person\\\/e4f2a890e079e0335c1cf66200edd937\"},\"headline\":\"Search at Intercom: Building our Elasticsearch cloud on AWS\",\"datePublished\":\"2021-04-13T16:34:43+00:00\",\"dateModified\":\"2021-05-12T13:24:24+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/building-elasticsearch-at-intercom\\\/\"},\"wordCount\":1627,\"publisher\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/building-elasticsearch-at-intercom\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/Hero-Elasticsearch-1800.jpg\",\"keywords\":[\"elasticsearch\",\"recruitment\"],\"articleSection\":[\"Engineering\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/building-elasticsearch-at-intercom\\\/\",\"url\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/building-elasticsearch-at-intercom\\\/\",\"name\":\"Search at Intercom: Building our Elasticsearch cloud on AWS - The Intercom Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/building-elasticsearch-at-intercom\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/building-elasticsearch-at-intercom\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/Hero-Elasticsearch-1800.jpg\",\"datePublished\":\"2021-04-13T16:34:43+00:00\",\"dateModified\":\"2021-05-12T13:24:24+00:00\",\"description\":\"Elasticsearch is a core technology at Intercom, and here's how we run our search infrastructure on AWS using EC2 hosts.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.intercom.com\\\/blog\\\/building-elasticsearch-at-intercom\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/building-elasticsearch-at-intercom\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/Hero-Elasticsearch-1800.jpg\",\"contentUrl\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/Hero-Elasticsearch-1800.jpg\",\"width\":1800,\"height\":842},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/\",\"name\":\"The Intercom Blog\",\"description\":\"Articles and Podcasts on Customer Service, AI and Automation, Product, and more\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#organization\",\"name\":\"The Intercom Blog\",\"url\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/08\\\/Intercom-logo-sq-black-trans.png\",\"contentUrl\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/08\\\/Intercom-logo-sq-black-trans.png\",\"width\":1000,\"height\":1000,\"caption\":\"The Intercom Blog\"},\"image\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/intercominc\",\"https:\\\/\\\/x.com\\\/intercom\",\"https:\\\/\\\/www.instagram.com\\\/intercom\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/2491343\",\"https:\\\/\\\/www.pinterest.ie\\\/intercom\\\/\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UCJG0MvLP03kyzzAkD-w98aQ\",\"https:\\\/\\\/en.wikipedia.org\\\/wiki\\\/Intercom_(company)\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#\\\/schema\\\/person\\\/e4f2a890e079e0335c1cf66200edd937\",\"name\":\"Andrej Blagojevi\u0107\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/0c0f4746fb6f602bcf7b9bdaa565109b754d1b0b76ce32a7300fd3f4c4ad4e0c?s=96&d=mm&r=pge573d43308c18df0979c7561c4a03c6e\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/0c0f4746fb6f602bcf7b9bdaa565109b754d1b0b76ce32a7300fd3f4c4ad4e0c?s=96&d=mm&r=pg\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/0c0f4746fb6f602bcf7b9bdaa565109b754d1b0b76ce32a7300fd3f4c4ad4e0c?s=96&d=mm&r=pg\",\"caption\":\"Andrej Blagojevi\u0107\"},\"url\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/author\\\/andrej_blagojevic\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Search at Intercom: Building our Elasticsearch cloud on AWS - The Intercom Blog","description":"Elasticsearch is a core technology at Intercom, and here's how we run our search infrastructure on AWS using EC2 hosts.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/","og_locale":"en_US","og_type":"article","og_title":"Search at Intercom: Building our Elasticsearch cloud on AWS","og_description":"Elasticsearch is a core technology at Intercom, and here's how we run our search infrastructure on AWS using EC2 hosts.","og_url":"https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/","og_site_name":"The Intercom Blog","article_publisher":"https:\/\/www.facebook.com\/intercominc","article_published_time":"2021-04-13T16:34:43+00:00","article_modified_time":"2021-05-12T13:24:24+00:00","og_image":[{"width":1800,"height":842,"url":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2021\/04\/Hero-Elasticsearch-1800.jpg","type":"image\/jpeg"}],"author":"Andrej Blagojevi\u0107","twitter_card":"summary_large_image","twitter_creator":"@intercom","twitter_site":"@intercom","twitter_misc":{"Written by":"Andrej Blagojevi\u0107","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/#article","isPartOf":{"@id":"https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/"},"author":{"name":"Andrej Blagojevi\u0107","@id":"https:\/\/www.intercom.com\/blog\/#\/schema\/person\/e4f2a890e079e0335c1cf66200edd937"},"headline":"Search at Intercom: Building our Elasticsearch cloud on AWS","datePublished":"2021-04-13T16:34:43+00:00","dateModified":"2021-05-12T13:24:24+00:00","mainEntityOfPage":{"@id":"https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/"},"wordCount":1627,"publisher":{"@id":"https:\/\/www.intercom.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/#primaryimage"},"thumbnailUrl":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2021\/04\/Hero-Elasticsearch-1800.jpg","keywords":["elasticsearch","recruitment"],"articleSection":["Engineering"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/","url":"https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/","name":"Search at Intercom: Building our Elasticsearch cloud on AWS - The Intercom Blog","isPartOf":{"@id":"https:\/\/www.intercom.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/#primaryimage"},"image":{"@id":"https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/#primaryimage"},"thumbnailUrl":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2021\/04\/Hero-Elasticsearch-1800.jpg","datePublished":"2021-04-13T16:34:43+00:00","dateModified":"2021-05-12T13:24:24+00:00","description":"Elasticsearch is a core technology at Intercom, and here's how we run our search infrastructure on AWS using EC2 hosts.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.intercom.com\/blog\/building-elasticsearch-at-intercom\/#primaryimage","url":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2021\/04\/Hero-Elasticsearch-1800.jpg","contentUrl":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2021\/04\/Hero-Elasticsearch-1800.jpg","width":1800,"height":842},{"@type":"WebSite","@id":"https:\/\/www.intercom.com\/blog\/#website","url":"https:\/\/www.intercom.com\/blog\/","name":"The Intercom Blog","description":"Articles and Podcasts on Customer Service, AI and Automation, Product, and more","publisher":{"@id":"https:\/\/www.intercom.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.intercom.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.intercom.com\/blog\/#organization","name":"The Intercom Blog","url":"https:\/\/www.intercom.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.intercom.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2019\/08\/Intercom-logo-sq-black-trans.png","contentUrl":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2019\/08\/Intercom-logo-sq-black-trans.png","width":1000,"height":1000,"caption":"The Intercom Blog"},"image":{"@id":"https:\/\/www.intercom.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/intercominc","https:\/\/x.com\/intercom","https:\/\/www.instagram.com\/intercom\/","https:\/\/www.linkedin.com\/company\/2491343","https:\/\/www.pinterest.ie\/intercom\/","https:\/\/www.youtube.com\/channel\/UCJG0MvLP03kyzzAkD-w98aQ","https:\/\/en.wikipedia.org\/wiki\/Intercom_(company)"]},{"@type":"Person","@id":"https:\/\/www.intercom.com\/blog\/#\/schema\/person\/e4f2a890e079e0335c1cf66200edd937","name":"Andrej Blagojevi\u0107","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/0c0f4746fb6f602bcf7b9bdaa565109b754d1b0b76ce32a7300fd3f4c4ad4e0c?s=96&d=mm&r=pge573d43308c18df0979c7561c4a03c6e","url":"https:\/\/secure.gravatar.com\/avatar\/0c0f4746fb6f602bcf7b9bdaa565109b754d1b0b76ce32a7300fd3f4c4ad4e0c?s=96&d=mm&r=pg","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/0c0f4746fb6f602bcf7b9bdaa565109b754d1b0b76ce32a7300fd3f4c4ad4e0c?s=96&d=mm&r=pg","caption":"Andrej Blagojevi\u0107"},"url":"https:\/\/www.intercom.com\/blog\/author\/andrej_blagojevic\/"}]}},"jetpack_featured_media_url":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2021\/04\/Hero-Elasticsearch-1800.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/posts\/25717","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/users\/475"}],"replies":[{"embeddable":true,"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/comments?post=25717"}],"version-history":[{"count":0,"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/posts\/25717\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/media\/25716"}],"wp:attachment":[{"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/media?parent=25717"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/category?post=25717"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/tags?post=25717"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/coauthors?post=25717"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}