{"id":312685,"xmlId":"com.intellij.bigdatatools","name":"Big Data Tools","description":"\n    <p>A plugin for data engineers and other specialists engaged with big data workloads. Installed in your favorite JetBrains IDE, Big Data Tools helps develop, visualize, debug, and monitor big data pipelines built in Scala, Python, and SQL.</p>\n    <p>\n      <a href=\"https://info.jetbrains.com/data-engineering-communication-center.html\">Subscribe to our News</a> |\n      <a href=\"https://blog.jetbrains.com/big-data-tools\">Read our Blog</a> |\n      <a href=\"https://slack-bdt.mau.jetbrains.com\">Join the Slack Community</a>\n    </p>\n    <br>\n\n    <h3>Use Big Data Tools for:</h3>\n    <dl>\n    <dd>- Exploratory analysis, visualization, and prototyping jobs in Zeppelin notebooks.</dd>\n    <dd>- Running and monitoring Spark or Flink jobs directly from your IDE.</dd>\n    <dd>- Working with Amazon EMR clusters.</dd>\n    <dd>- Viewing big data binary files, such as CSV, Parquet, ORC, and Avro.</dd>\n    <dd>- Producing and consuming messages with Kafka.</dd>\n    <dd>- Previewing Hive Metastore databases.</dd>\n    <dd>- Getting insights about your Hadoop environment.</dd>\n    </dl>\n\n    <h3>Built-in tools and integrations:</h3>\n    <dl>\n    <dd>- <b>Supported languages:</b> Scala, Python, SQL.</dd>\n    <dd>- <b>Notebooks:</b> Zeppelin.</dd>\n    <dd>- <b>Monitoring:</b> Hadoop, Kafka, Spark, Hive Metastore, Flink, AWS Glue.</dd>\n    <dd>- <b>Remote file storages:</b> AWS S3, Google Cloud Storage, Microsoft Azure, Tencent Cloud Object Storage (COS), DigitalOcean Spaces, Alibaba OSS, Hadoop Distributed File System (HDFS), and more.</dd>\n    <dd>- <b>File systems:</b> HDFS, Local, SFTP.</dd>\n    <dd>- <b>Data processing platforms:</b> AWS EMR.</dd>\n    </dl>\n    \n<h2>Getting started</h2>\n* These instructions link to the Big Data Tools <a href=\"https://www.jetbrains.com/help/idea/big-data-tools-support.html\">documentation for IntelliJ IDEA</a>. If you use the plugin with a different JetBrains IDE, please use one of the links below instead:\n<br><br>\n<a href=\"https://www.jetbrains.com/help/pycharm/big-data-tools-support.html\">PyCharm</a> | <a href=\"https://www.jetbrains.com/help/dataspell/big-data-tools-support.html\">DataSpell</a> | <a href=\"https://www.jetbrains.com/help/datagrip/big-data-tools-support.html\">DataGrip</a>\n<br><br><b>To start using Big Data Tools in IntelliJ IDEA:</b><ol>\n<li><a href=\"https://www.jetbrains.com/help/idea/managing-plugins.html\">Install</a> the Big Data Tools plugin.\n</li><li>Install the required language plugins (<a href=\"https://plugins.jetbrains.com/plugin/1347-scala\">Scala</a> or <a href=\"https://plugins.jetbrains.com/plugin/631-python\">Python</a>).\n</li><li>Create a <a href=\"https://www.jetbrains.com/help/idea/new-project-wizard.html\">new project</a> in your IDE.\n</li><li><a href=\"https://www.jetbrains.com/help/idea/big-data-tools-configuration.html#connection\">Connect</a> to a particular server, storage, or service.\n</li><li>Voila! You’re ready to start working on your project.</li></ol>\n","organization":"JetBrains s.r.o.","tags":["Tools Integration","Cloud","Data Science","Viewer"],"vendor":"JetBrains","version":"231.8109.175","notes":"\n<h2>Changes in 2023.1</h2>\n<h3>Features</h3>\n<h4>Zeppelin</h4>\n<ul>\n   <li>More Extract options. In addition to extracting a selected Scala code into a separate method or class, you can now:\n   <ul>\n      <li>Extract all Scala paragraphs of a Zeppelin note</li>\n      <li>Extract Scala code from a specific paragraph</li>\n   </ul>\n   </li>\n   <li>Interpreters can be refreshed globally, not only per notebook</li>\n   <li>Zeppelin dependencies, interpreter settings, and interpreter bindings are now all merged into a single dialog, \"Dependencies and Interpreter Settings\"</li>\n   <li>Add completion for columns names in PySpark</li>\n</ul>\n\n<h4>Hive Metastore</h4>\n<ul>\n   <li>Hive version v2 is supported now</li>\n</ul>\n\n<h4>Kafka</h4>\n<ul>\n   <li>Supported connections to Kafka Schema Registry</li>\n   <li>Enabled SSH tunneling for Kafka Schema Registry</li>\n   <li>Supported AWS IAM authentication</li>\n</ul>\n\n<h4>Kerberos</h4>\n<ul>\n   <li>Supported connection through Kerberos for\n      <ul>\n        <li>Kafka</li>\n        <li>Hive Metastore</li>\n        <li>HDFS</li>\n      </ul>\n   </li>\n</ul>\n\n<h4>Remote File Storages</h4>\n<ul>\n   <li>Extended support for all clouds in Big Data Tools:\n      <ul>\n         <li>Universal meta info</li>\n         <li>File versioning</li>\n         <li>Showing and editing tags for objects and buckets</li>\n         <li>Showing cross origin replication status (CORS)</li>\n      </ul>\n   </li>\n   <li>Filtering buckets by name in storages</li>\n   <li>The \"Create directory\" action allows now creating nested directories</li>\n   <li>Added new options to the Copy File Path action:\n       <ul>\n          <li>Copy Key Path</li>\n          <li>Copy Presigned Path</li>\n       </ul>\n   </li>\n   <li>The \"Navigate to\" action is available now for HDFS, Local, and SFTP connections</li>\n</ul>\n\n<h4>Big Data Tools ToolWindow</h4>\n<ul>\n   <li>Multiple connections can be disabled in the Big Data Tools tool window</li>\n</ul>\n\n<h4>Authorization</h4>\n<ul>\n   <li>AWS authorizations are shared between the following connections:\n       <ul>\n           <li>AWS S3</li>\n           <li>AWS Glue</li>\n           <li>AWS EMR</li>\n       </ul>\n   </li>\n   <li>Azure: The following types of connections by:\n       <ul>\n          <li>Azure Account\n             <ul>\n                <li>OAuth 2.0</li>\n                <li>Device code</li>\n                <li>Azure CLI</li>\n             </ul>\n          </li>\n       </ul>\n   </li>\n   <li>Google Cloud Storage: Support CLI OAuth</li>\n</ul>\n\n<h3>Important Fixes</h3>\n<h4>Remote File Storage</h4>\n<ul>\n   <li>We do not try to reconnect now in case of authentication failure</li>\n</ul>\n\n<h4>Hive Metastore</h4>\n<ul>\n   <li>“thrift://” is now supported in URL</li>\n</ul>\n\n<h4>Parquet Files</h4>\n<ul>\n   <li>You can now open Parquet files that use zstd, Brotli, and LZ4 compression</li>\n</ul>\n\n<p>The full release notes are available <a href=\"https://youtrack.jetbrains.com/issues?q=project:%20%7BBig%20Data%20IDE%7D%20Milestone:%202023.1%20State:%20Fixed,%20Verified,%20%7BWithout%20verification%7D%20sort%20by:%20project,%20Type,%20votes\">here</a>.</p>\n","dependencies":["com.intellij.modules.ultimate","intellij.grid.impl"],"optionalDependencies":["org.intellij.scala","org.jetbrains.plugins.gradle","org.jetbrains.idea.maven","org.jetbrains.kotlin","com.intellij.database","org.intellij.plugins.markdown","com.jetbrains.sh","org.jetbrains.plugins.remote-run","Pythonid","PythonCore","com.jetbrains.plugins.webDeployment"],"since":"231.8109","until":"231.*","size":104816609}